Oct 29 23:12:31.084157 ip-10-0-139-224 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 23:12:31.474551 ip-10-0-139-224 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 23:12:31.474551 ip-10-0-139-224 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 23:12:31.474551 ip-10-0-139-224 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 23:12:31.474551 ip-10-0-139-224 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 23:12:31.477950 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477862    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.477950 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477950    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477954    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477958    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477961    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477966    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477968    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477972    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477976    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477978    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477981    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477984    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477986    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477989    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.477996    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478000    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478003    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478005    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478008    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478011    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478014    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.478016 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478016    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478020    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478022    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478025    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478028    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478031    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478034    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478036    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478039    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478041    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478044    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478046    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478049    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478052    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478054    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478057    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478059    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478062    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478064    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478067    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.478489 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478069    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478072    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478074    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478077    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478079    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478082    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478085    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478088    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478090    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478093    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478095    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478099    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478101    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478104    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478108    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478110    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478113    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478115    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478118    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478120    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.479005 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478123    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478125    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478128    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478131    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478134    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478137    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478139    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478142    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478144    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478147    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478149    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478152    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478155    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478157    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478160    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478162    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478165    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478169    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478171    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478174    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.479471 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478177    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478180    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478182    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478185    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478187    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478190    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478193    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478196    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478200    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478204    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478208    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478211    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478214    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478217    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478220    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478224    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478228    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478230    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.479993 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478233    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478236    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478238    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478242    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478244    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478247    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.478250    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479709    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479718    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479722    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479725    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479728    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479731    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479735    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479740    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479745    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479748    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479751    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479754    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.480413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479756    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479759    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479762    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479764    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479767    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479769    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479772    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479775    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479777    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479780    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479782    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479785    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479788    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479790    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479793    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479795    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479798    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479800    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479803    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.481004 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479807    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479810    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479812    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479815    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479817    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479820    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479822    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479825    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479827    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479831    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479834    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479836    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479839    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479841    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479844    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479847    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479849    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479852    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479854    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479857    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.481449 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479859    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479862    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479864    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479867    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479870    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479873    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479898    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479903    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479907    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479910    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479912    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479915    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479918    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479920    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479923    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479925    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479928    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479931    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479933    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479935    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.481941 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479938    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479941    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479945    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479948    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479951    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479953    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479956    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479959    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479961    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479964    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479966    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479969    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479971    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479974    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479977    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479979    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479982    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479984    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479987    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479992    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.482448 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479996    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.479999    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480002    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480005    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480007    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480010    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480013    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480015    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480018    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480020    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480023    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480026    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480030    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480034    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.480037    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481387    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481394    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481398    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481401    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.482933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481404    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481407    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481410    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481412    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481415    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481418    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481421    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481423    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481428    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481432    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481436    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481439    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481442    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481445    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481448    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481451    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481453    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481456    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481459    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481461    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.483385 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481464    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481467    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481469    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481472    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481475    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481478    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481482    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481484    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481488    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481491    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481493    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481495    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481498    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481500    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481503    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481506    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481508    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481510    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481513    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481515    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.483848 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481518    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481521    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481523    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481526    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481528    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481531    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481533    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481536    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481538    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481541    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481543    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481546    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481548    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481551    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481554    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481556    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481561    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481564    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481567    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481570    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.484333 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481572    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481575    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481578    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481581    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481583    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481585    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481588    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481590    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481592    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481595    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481597    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481600    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481603    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481606    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481608    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481611    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481613    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481616    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481619    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.484792 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481621    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481623    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481626    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481628    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481631    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481633    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481635    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481638    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481640    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481643    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481645    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481648    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481650    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481653    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481655    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481658    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481661    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481665    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481667    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.485268 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481669    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481672    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481674    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.481677    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482599    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482614    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482622    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482627    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482632    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482635    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482639    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482648    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482651    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482654    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482658    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482661    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482665    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482668    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482671    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482674    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482677    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482680    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482684    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482687    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 23:12:31.485710 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482690    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482693    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482697    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482701    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482704    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482708    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482712    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482715    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482718    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482721    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482724    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482727    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482731    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482736    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482739    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482742    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482746    2582 flags.go:64] FLAG: --enable-server="true"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482749    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482754    2582 flags.go:64] FLAG: --event-burst="100"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482757    2582 flags.go:64] FLAG: --event-qps="50"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482760    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482764    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482767    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482771    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 23:12:31.486263 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482776    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482779    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482782    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482786    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482790    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482793    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482796    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482799    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482802    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482805    2582 flags.go:64] FLAG: --feature-gates=""
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482809    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482813    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482817    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482820    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482824    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482827    2582 flags.go:64] FLAG: --help="false"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482830    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482833    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482837    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482840    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482844    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482849    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482852    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 23:12:31.487524 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482855    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482858    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482861    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482864    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482867    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482870    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482873    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482888    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482891    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482896    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482899    2582 flags.go:64] FLAG: --lock-file=""
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482902    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482905    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482908    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482914    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482917    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482920    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482923    2582 flags.go:64] FLAG: --logging-format="text"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482926    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482930    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482933    2582 flags.go:64] FLAG: --manifest-url=""
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482936    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482941    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482944    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482949    2582 flags.go:64] FLAG: --max-pods="110"
Oct 29 23:12:31.488116 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482952    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482955    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482958    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482961    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482964    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482969    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482972    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482980    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482984    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482987    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482990    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482993    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482996    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.482999    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483003    2582 flags.go:64] FLAG: --port="10250"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483006    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483010    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c129f8b72838dc6a"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483014    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483017    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483020    2582 flags.go:64] FLAG: --register-node="true"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483023    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483027    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483030    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483033    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483036    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483040    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 23:12:31.488691 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483044    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483047    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483050    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483053    2582 flags.go:64] FLAG: --runonce="false"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483056    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483059    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483063    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483067    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483070    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483073    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483076    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483081    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483084    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483087    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483091    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483094    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483097    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483100    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483103    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483109    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483112    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483115    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483143    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483147    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483150    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 23:12:31.489362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483153    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483157    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483160    2582 flags.go:64] FLAG: --v="2"
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483165    2582 flags.go:64] FLAG: --version="false"
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483169    2582 flags.go:64] FLAG: --vmodule=""
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483173    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483177    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483316    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483320    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483323    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483326    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483328    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483331    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483334    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483337    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483340    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483343    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483346    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483349    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483352    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483355    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.490112 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483357    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483360    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483363    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483365    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483368    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483370    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483373    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483376    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483379    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483382    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483384    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483387    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483390    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483392    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483395    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483398    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483401    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483403    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483406    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483408    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.490943 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483411    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483413    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483417    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483420    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483423    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483426    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483430    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483433    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483435    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483439    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483442    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483444    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483447    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483450    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483453    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483456    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483458    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483461    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483464    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.491460 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483466    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483469    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483472    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483475    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483477    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483480    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483482    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483486    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483490    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483493    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483496    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483499    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483502    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483504    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483507    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483509    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483512    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483514    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483517    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483519    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.491933 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483522    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483525    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483529    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483531    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483534    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483536    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483540    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483543    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483546    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483548    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483551    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483553    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483556    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483559    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483561    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483564    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483566    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483569    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483572    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.492413 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483574    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483577    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483579    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483582    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483584    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483587    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483590    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483592    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483595    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483597    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483599    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483602    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483605    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483607    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.483634    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483668    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.492869 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483675    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483678    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483681    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483684    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483687    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483689    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483692    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483695    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483698    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483700    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483703    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483706    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483708    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483711    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483713    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483716    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483719    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483722    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483724    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483726    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.493294 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483729    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483731    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483734    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483737    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483739    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483742    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483745    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483749    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483752    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483754    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483757    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483763    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483767    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483770    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483773    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483776    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483778    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483781    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483784    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.493758 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483787    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483789    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483792    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483795    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483797    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483800    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483803    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483805    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483807    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483810    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483813    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483816    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483820    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483823    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483825    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483828    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483830    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483833    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483835    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483838    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.494237 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483840    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483843    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483846    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483848    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483850    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483855    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483859    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483861    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483864    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483866    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483869    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483871    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483888    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483893    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483898    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483902    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483906    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483910    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483912    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.494731 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483915    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483918    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483920    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483923    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483925    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483928    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483930    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483933    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483935    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483938    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483940    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483943    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483945    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483948    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483950    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483953    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483955    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483958    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483961    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483965    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.495209 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483968    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483971    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483973    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483976    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483978    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483981    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.483983    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.492164    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.492182    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.492205    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.492210    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492274    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492280    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492284    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492287    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492293    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492295    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492298    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.495684 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492300    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492303    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492305    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492308    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492311    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492313    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492316    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492319    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492321    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492324    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492327    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492329    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492332    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492335    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492337    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492340    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492344    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492348    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492351    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.496222 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492355    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492358    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492362    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492364    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492367    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492370    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492372    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492375    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492377    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492380    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492382    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492385    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492387    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492390    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492392    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492395    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492398    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492400    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492403    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492406    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.496666 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492408    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492412    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492415    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492418    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492421    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492424    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492428    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492432    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492435    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492438    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492441    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492444    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492447    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492450    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492453    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492457    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492460    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492462    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.497163 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492465    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492468    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492471    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492473    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492476    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492478    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492481    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492483    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492486    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492489    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492491    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492494    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492497    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492499    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492503    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492506    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492508    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492511    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492513    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492516    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.497575 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492519    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492521    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492523    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492526    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492528    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492531    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492533    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492536    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492539    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492542    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492544    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492547    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492550    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492552    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492555    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492557    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492560    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492563    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492565    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492568    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.498053 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492570    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.492598    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492741    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492746    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492750    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492753    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492755    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492760    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492763    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492766    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492768    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492771    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492773    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492776    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492779    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.498533 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492781    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492784    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492787    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492789    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492792    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492794    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492797    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492800    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492802    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492805    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492807    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492810    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492812    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492815    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492817    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492820    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492822    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492825    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492827    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492830    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.498890 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492832    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492835    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492837    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492840    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492842    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492845    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492848    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492850    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492853    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492855    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492858    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492861    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492863    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492866    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492869    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492871    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492889    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492892    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492895    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.499364 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492897    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492900    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492902    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492905    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492908    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492910    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492913    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492916    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492918    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492921    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492923    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492926    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492928    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492931    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492934    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492936    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492938    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492941    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492944    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.500338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492948    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492950    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492953    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492956    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492958    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492960    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492963    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492966    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492969    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492971    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492975    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492978    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492981    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492984    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492987    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492990    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492993    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492996    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.492999    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493001    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.500778 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493004    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493007    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493009    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493012    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493014    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493017    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493019    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493022    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493025    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493029    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493031    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493034    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493037    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493041    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493044    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.493068    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 23:12:31.501306 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493099    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493103    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493106    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493109    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493112    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493114    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493117    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493120    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493123    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493125    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493128    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493131    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493133    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493136    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493139    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493141    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493144    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493146    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493149    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493151    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:12:31.501697 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493154    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493156    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493162    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493164    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493167    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493170    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493172    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493175    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493177    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493180    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493183    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493186    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493188    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493191    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493193    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493196    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493198    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493201    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493204    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493206    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:12:31.502206 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493209    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493211    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493214    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493216    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493218    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493221    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493223    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493226    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493229    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493233    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493235    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493238    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493240    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493243    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493247    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493250    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493252    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493255    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493258    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:12:31.502683 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493261    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493263    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493266    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493268    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493271    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493274    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493276    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493279    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493282    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493284    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493287    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493289    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493292    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493295    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493297    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493300    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493304    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493307    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493311    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:12:31.503213 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493315    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493318    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493320    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493324    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493326    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493329    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493332    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493334    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493338    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493340    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493343    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493346    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493348    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493351    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493354    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493356    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493359    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493364    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493368    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:12:31.503699 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493371    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493374    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493377    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493379    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493382    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493385    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493387    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493390    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:31.493392    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.493452    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.496841    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.497769    2582 server.go:1014] "Starting client certificate rotation"
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.497865    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.497915    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.500416    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 23:12:31.504296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.501720    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 23:12:31.514459 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.514439    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 29 23:12:31.517487 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.517474    2582 log.go:25] "Validated CRI v1 image API"
Oct 29 23:12:31.518280 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.518256    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 23:12:31.522325 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.522298    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 807f0546-93dd-4376-81f1-2270604a75c1:/dev/nvme0n1p3 f74100c9-14a0-46ff-aea3-8ff38ce07abd:/dev/nvme0n1p4]
Oct 29 23:12:31.522422 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.522323    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 23:12:31.528232 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.528111    2582 manager.go:217] Machine: {Timestamp:2026-10-29 23:12:31.526672965 +0000 UTC m=+0.330698258 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099851 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21633e0ed6a78a235cdd62784fe1e5 SystemUUID:ec21633e-0ed6-a78a-235c-dd62784fe1e5 BootID:fe0e5392-2ca4-4e8e-a139-24b98ae44d9f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:99:e3:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:99:e3:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:fa:d9:60:51:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 23:12:31.528232 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.528224    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 23:12:31.528363 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.528333    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 23:12:31.532031 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.532004    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 23:12:31.532161 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.532033    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 23:12:31.532210 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.532165    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 23:12:31.532210 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.532173    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 23:12:31.532262 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.532208    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 23:12:31.532262 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.532228    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 23:12:31.533054 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.533044    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 23:12:31.536519 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.536507    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 23:12:31.536630 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.536621    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 23:12:31.537294 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.537285    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 23:12:31.537328 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.537298    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 23:12:31.537328 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.537309    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 23:12:31.537328 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.537318    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 29 23:12:31.537328 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.537326    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 23:12:31.537654 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.537643    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 23:12:31.539556 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.539538    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 23:12:31.544312 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.544299    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 23:12:31.545140 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545127    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 23:12:31.545180 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545159    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 23:12:31.545330 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545312    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 23:12:31.545432 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545342    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 23:12:31.545432 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545360    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 23:12:31.545432 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545374    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 23:12:31.545432 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545390    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 23:12:31.545432 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545399    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 23:12:31.545908 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545897    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 23:12:31.545964 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545910    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 23:12:31.545964 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545923    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 23:12:31.545964 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545932    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 23:12:31.545964 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545948    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 23:12:31.545964 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.545962    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 23:12:31.546654 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.546643    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 23:12:31.546708 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.546658    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 23:12:31.550543 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.550527    2582 server.go:1263] "Started kubelet"
Oct 29 23:12:31.550627 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.550598    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 23:12:31.550720 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.550642    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 23:12:31.550787 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.550738    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 23:12:31.551358 ip-10-0-139-224 systemd[1]: Started Kubernetes Kubelet.
Oct 29 23:12:31.552001 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.551985    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 23:12:31.553350 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.553337    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 23:12:31.556965 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.556917    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 23:12:31.559261 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.559247    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 23:12:31.559698 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.559679    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 23:12:31.559835 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.559795    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:31.560348 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.560327    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 23:12:31.560348 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.560349    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 23:12:31.560488 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.560333    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 23:12:31.560488 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.560432    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 23:12:31.560488 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.560438    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 23:12:31.561202 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.561169    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 23:12:31.561285 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.561204    2582 factory.go:55] Registering systemd factory
Oct 29 23:12:31.561285 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.561215    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 29 23:12:31.562820 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.562795    2582 factory.go:153] Registering CRI-O factory
Oct 29 23:12:31.562820 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.562822    2582 factory.go:223] Registration of the crio container factory successfully
Oct 29 23:12:31.562965 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.562840    2582 factory.go:103] Registering Raw factory
Oct 29 23:12:31.562965 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.562854    2582 manager.go:1199] Started watching for new ooms in manager
Oct 29 23:12:31.564444 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.564424    2582 manager.go:319] Starting recovery of all containers
Oct 29 23:12:31.569284 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.569233    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 23:12:31.575043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.574946    2582 manager.go:324] Recovery completed
Oct 29 23:12:31.579545 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.579530    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:12:31.581402 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.581386    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:12:31.581487 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.581412    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:12:31.581487 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.581422    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:12:31.581918 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.581905    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 29 23:12:31.581918 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.581917    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 23:12:31.582022 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.581938    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 23:12:31.582022 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.581944    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:12:31.583281 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.583264    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:12:31.583340 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.583292    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:12:31.583340 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.583302    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:12:31.583981 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.583970    2582 policy_none.go:50] "Start"
Oct 29 23:12:31.584025 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.583984    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 23:12:31.584025 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.583994    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 23:12:31.585834 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.585817    2582 policy_none.go:44] "Start"
Oct 29 23:12:31.587161 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.587142    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-224.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 23:12:31.587228 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.587159    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 23:12:31.587981 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.587147    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-224.ec2.internal.18e322fe2459e94c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-224.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-224.ec2.internal,},FirstTimestamp:2026-10-29 23:12:31.550499148 +0000 UTC m=+0.354524440,LastTimestamp:2026-10-29 23:12:31.550499148 +0000 UTC m=+0.354524440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-224.ec2.internal,}"
Oct 29 23:12:31.588762 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.588744    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 23:12:31.588860 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.588823    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z57sw"
Oct 29 23:12:31.601036 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.601017    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z57sw"
Oct 29 23:12:31.604381 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.604349    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 23:12:31.604381 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.604353    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 23:12:31.604590 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.604505    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-224.ec2.internal.18e322fe26316be3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-224.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-224.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-224.ec2.internal,},FirstTimestamp:2026-10-29 23:12:31.581400035 +0000 UTC m=+0.385425326,LastTimestamp:2026-10-29 23:12:31.581400035 +0000 UTC m=+0.385425326,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-224.ec2.internal,}"
Oct 29 23:12:31.613381 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.613351    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 23:12:31.633545 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.633533    2582 manager.go:341] "Starting Device Plugin manager"
Oct 29 23:12:31.633626 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.633565    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 23:12:31.633626 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.633573    2582 server.go:87] "Starting device plugin registration server"
Oct 29 23:12:31.634298 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.633991    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 23:12:31.634298 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.634011    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 23:12:31.634298 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.634286    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 23:12:31.636008 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.634359    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 23:12:31.636008 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.634368    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 23:12:31.636357 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.636335    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 23:12:31.636488 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.636388    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:31.665567 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.665547    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:12:31.734471 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.734402    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:12:31.735768 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.735375    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:12:31.735768 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.735404    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:12:31.735768 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.735414    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:12:31.735768 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.735441    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.754646 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.754625    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.754727 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.754656    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-224.ec2.internal\": node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:31.758649 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.758626    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 23:12:31.758649 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.758650    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 23:12:31.758753 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.758667    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 23:12:31.758753 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.758696    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 23:12:31.777352 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.777332    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:12:31.795098 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.795078    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:31.859233 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.859210    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"]
Oct 29 23:12:31.859343 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.859288    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:12:31.860216 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.860201    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:12:31.860297 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.860257    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:12:31.860297 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.860273    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:12:31.861343 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.861315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.861343 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.861340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.861977 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.861964    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:12:31.862141 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862128    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.862175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862157    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:12:31.862751 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862735    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:12:31.862833 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862761    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:12:31.862833 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862770    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:12:31.862833 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862736    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:12:31.862966 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862833    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:12:31.862966 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.862850    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:12:31.864716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.864703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.864756 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.864728    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:12:31.865422 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.865407    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:12:31.865492 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.865439    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:12:31.865492 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.865451    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:12:31.885341 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.885324    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-224.ec2.internal\" not found" node="ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.886901 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.886869    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-224.ec2.internal\" not found" node="ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.895659 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.895643    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:31.961788 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.961760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.961901 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.961792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.961901 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.961819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88f04efcc82b4a40de30b52c3590d78c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" (UID: \"88f04efcc82b4a40de30b52c3590d78c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.961901 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.961894    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.962020 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:31.961900    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:31.996170 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:31.996123    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:32.062300 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.062276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88f04efcc82b4a40de30b52c3590d78c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" (UID: \"88f04efcc82b4a40de30b52c3590d78c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:32.062371 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.062331    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88f04efcc82b4a40de30b52c3590d78c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" (UID: \"88f04efcc82b4a40de30b52c3590d78c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:32.096905 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.096872    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:32.189363 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.189339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:32.191522 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.191503    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:32.197615 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.197599    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:32.298280 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.298073    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:32.398343 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.398314    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:12:32.456932 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.456905    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:12:32.460083 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.460056    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:32.485561 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.485540    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 23:12:32.486802 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.486789    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:32.497545 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.497528    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 23:12:32.497672 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.497654    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 23:12:32.497728 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.497702    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 23:12:32.497765 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.497712    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 23:12:32.497765 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.497734    2582 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a652e1dff22fb4b1682347fb8adf3ec0-fbff9162e07032f1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.139.224:54560->18.116.0.247:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:12:32.539518 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.539496    2582 apiserver.go:52] "Watching apiserver"
Oct 29 23:12:32.560091 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.560046    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 23:12:32.605567 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.605534    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:07:32 +0000 UTC" deadline="2028-04-23 19:00:36.116502017 +0000 UTC"
Oct 29 23:12:32.605567 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.605562    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13003h48m3.510942738s"
Oct 29 23:12:32.632373 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.632338    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 23:12:32.632667 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.632647    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-44xj8","openshift-network-diagnostics/network-check-target-b97hf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn","openshift-image-registry/node-ca-qwc9k","kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4bvzv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal","openshift-multus/multus-2kg5q","openshift-multus/multus-additional-cni-plugins-mzt69","openshift-multus/network-metrics-daemon-bc5z8","kube-system/global-pull-secret-syncer-mkj2j","openshift-dns/node-resolver-zx5vd","openshift-network-operator/iptables-alerter-glknz","openshift-ovn-kubernetes/ovnkube-node-dv7ws"]
Oct 29 23:12:32.635310 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.635288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:32.635421 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.635394    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:32.637475 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.637455    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.637567 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.637522    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:32.639842 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.639826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.642059 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.642035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.644534 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.644514    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.646884 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.646865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.649271 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.649253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.652633 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.652610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.655186 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.655162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:32.655292 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.655241    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:32.658280 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.658254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:32.661479 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.661079    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663181    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663211    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663258    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663285    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663318    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663335    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.664000 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.663574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.665777 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.665758    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 23:12:32.665895 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.665883    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kcr8p\""
Oct 29 23:12:32.665965 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.665928    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vsvpd\""
Oct 29 23:12:32.665965 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.665932    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h2str\""
Oct 29 23:12:32.665965 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.665932    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w5p5k\""
Oct 29 23:12:32.666644 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.666699 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96cf9d61-5fb4-45ed-b68b-868a6738f12c-serviceca\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.666699 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666666    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-netns\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.666699 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-kubelet\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.666846 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-bin\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.666846 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f643dab0-17a4-410b-b56b-f278862fa237-host-slash\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.666846 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:32.667024 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-cni-binary-copy\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.667024 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-dbus\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.667024 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqjq\" (UniqueName: \"kubernetes.io/projected/96cf9d61-5fb4-45ed-b68b-868a6738f12c-kube-api-access-nzqjq\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.667024 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.666979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-modprobe-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.667024 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fd66\" (UniqueName: \"kubernetes.io/projected/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-kube-api-access-8fd66\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-hostroot\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-multus-certs\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667093    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-netns\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-ovn\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-config\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-systemd\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqr78\" (UniqueName: \"kubernetes.io/projected/2e327984-d0fc-4330-a933-f104718bce7c-kube-api-access-tqr78\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f643dab0-17a4-410b-b56b-f278862fa237-iptables-alerter-script\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.667227 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxp2\" (UniqueName: \"kubernetes.io/projected/d9a2e31a-049d-48d2-979c-8557f7c239f2-kube-api-access-dfxp2\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-netd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-script-lib\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbtb\" (UniqueName: \"kubernetes.io/projected/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-kube-api-access-rgbtb\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-var-lib-kubelet\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-tuned\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-etc-kubernetes\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-sys-fs\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.667584 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cf9d61-5fb4-45ed-b68b-868a6738f12c-host\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-systemd-units\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-kubernetes\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-tmp\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-socket-dir-parent\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-daemon-config\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e02182-3b19-41c8-9040-f75a4254187b-tmp-dir\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-registration-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-env-overrides\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prn7r\" (UniqueName: \"kubernetes.io/projected/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-kube-api-access-prn7r\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667958    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysconfig\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.667986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-kubelet-config\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-node-log\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668047    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-cnibin\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf7f273-a82a-40a5-811c-efb7281d1db6-agent-certs\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:32.668175 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-multus\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89e02182-3b19-41c8-9040-f75a4254187b-hosts-file\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668182    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-socket-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-device-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-system-cni-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-k8s-cni-cncf-io\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94g2q\" (UniqueName: \"kubernetes.io/projected/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kube-api-access-94g2q\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-run\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcz9\" (UniqueName: \"kubernetes.io/projected/89e02182-3b19-41c8-9040-f75a4254187b-kube-api-access-kgcz9\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668470    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-slash\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-log-socket\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-sys\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-host\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.668716 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-system-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-var-lib-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-kubelet\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-conf-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-etc-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-conf\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-lib-modules\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-bin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-systemd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.668993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf7f273-a82a-40a5-811c-efb7281d1db6-konnectivity-ca\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.669020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.669040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.669304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.669058    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpflj\" (UniqueName: \"kubernetes.io/projected/f643dab0-17a4-410b-b56b-f278862fa237-kube-api-access-lpflj\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.669885 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.669078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-cnibin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.675964 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.675935    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 23:12:32.676043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.676004    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nmnl7"
Oct 29 23:12:32.760839 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.760821    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 23:12:32.767202 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:32.767162    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f04efcc82b4a40de30b52c3590d78c.slice/crio-3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e WatchSource:0}: Error finding container 3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e: Status 404 returned error can't find the container with id 3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e
Oct 29 23:12:32.767440 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:32.767422    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410469b0f63ac5a751ec52b1c6fe6043.slice/crio-80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696 WatchSource:0}: Error finding container 80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696: Status 404 returned error can't find the container with id 80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696
Oct 29 23:12:32.769762 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e02182-3b19-41c8-9040-f75a4254187b-tmp-dir\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.769829 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-registration-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.769829 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-env-overrides\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.769829 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prn7r\" (UniqueName: \"kubernetes.io/projected/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-kube-api-access-prn7r\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.769829 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysconfig\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-kubelet-config\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-node-log\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769863    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-registration-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-cnibin\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769940    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf7f273-a82a-40a5-811c-efb7281d1db6-agent-certs\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769958    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-node-log\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.769981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-multus\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89e02182-3b19-41c8-9040-f75a4254187b-hosts-file\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.770077 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-socket-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-device-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770115    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-multus\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-system-cni-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770170    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-system-cni-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770176    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89e02182-3b19-41c8-9040-f75a4254187b-hosts-file\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-k8s-cni-cncf-io\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.770258    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94g2q\" (UniqueName: \"kubernetes.io/projected/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kube-api-access-94g2q\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-run\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.770330    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs podName:1d0b2a03-5d09-4c44-bec0-ec66bdea6781 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:33.270300795 +0000 UTC m=+2.074326094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs") pod "network-metrics-daemon-bc5z8" (UID: "1d0b2a03-5d09-4c44-bec0-ec66bdea6781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770336    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-run\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcz9\" (UniqueName: \"kubernetes.io/projected/89e02182-3b19-41c8-9040-f75a4254187b-kube-api-access-kgcz9\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770372    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770468    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysconfig\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.770542 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770505    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770552    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-k8s-cni-cncf-io\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770374    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-device-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770646    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770737    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770665    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770768    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770810    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-socket-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770858    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770879    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770933    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770985    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770994    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770760    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771039    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771067    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.770933    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771086    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.771125    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771092    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-slash\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771181    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-kubelet-config\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.771329 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.771208    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret podName:d9192aec-64e7-4d0f-8066-e236dc75b87d nodeName:}" failed. No retries permitted until 2026-10-29 23:12:33.271185677 +0000 UTC m=+2.075210966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret") pod "global-pull-secret-syncer-mkj2j" (UID: "d9192aec-64e7-4d0f-8066-e236dc75b87d") : object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-log-socket\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771244    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-cnibin\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771257    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-slash\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-sys\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771286    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-log-socket\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-host\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771314    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771341    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-sys\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-system-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-var-lib-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771400    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-host\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771433    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-system-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771446    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-var-lib-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-kubelet\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771500    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-conf-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771539    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-kubelet\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-etc-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-etc-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-conf\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772020    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-env-overrides\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-lib-modules\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.771598    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-conf-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e02182-3b19-41c8-9040-f75a4254187b-tmp-dir\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-bin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772352    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-conf\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-systemd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772405    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-systemd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772461    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-bin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772463    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-lib-modules\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.773955 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf7f273-a82a-40a5-811c-efb7281d1db6-konnectivity-ca\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpflj\" (UniqueName: \"kubernetes.io/projected/f643dab0-17a4-410b-b56b-f278862fa237-kube-api-access-lpflj\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-cnibin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.772831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96cf9d61-5fb4-45ed-b68b-868a6738f12c-serviceca\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-netns\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-kubelet\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-bin\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773240    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f643dab0-17a4-410b-b56b-f278862fa237-host-slash\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-cni-binary-copy\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-dbus\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqjq\" (UniqueName: \"kubernetes.io/projected/96cf9d61-5fb4-45ed-b68b-868a6738f12c-kube-api-access-nzqjq\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773667    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-modprobe-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fd66\" (UniqueName: \"kubernetes.io/projected/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-kube-api-access-8fd66\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:32.774927 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.773973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-hostroot\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-multus-certs\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774157    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5wvkd\""
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774159    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96xtz\""
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-netns\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-ovn\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-config\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-systemd\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqr78\" (UniqueName: \"kubernetes.io/projected/2e327984-d0fc-4330-a933-f104718bce7c-kube-api-access-tqr78\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774618    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774873    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fjkb\""
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775119    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfk9z\""
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.774617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f643dab0-17a4-410b-b56b-f278862fa237-iptables-alerter-script\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxp2\" (UniqueName: \"kubernetes.io/projected/d9a2e31a-049d-48d2-979c-8557f7c239f2-kube-api-access-dfxp2\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-netd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-script-lib\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbtb\" (UniqueName: \"kubernetes.io/projected/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-kube-api-access-rgbtb\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-var-lib-kubelet\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-tuned\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.776129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-etc-kubernetes\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-sys-fs\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cf9d61-5fb4-45ed-b68b-868a6738f12c-host\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-systemd-units\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-kubernetes\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-tmp\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-socket-dir-parent\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.777185 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.775797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-daemon-config\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.777705 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.777323    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sw5tb\""
Oct 29 23:12:32.777705 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.777514    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-netd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.777705 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.777534    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-systemd-units\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.777705 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.777611    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-sys-fs\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.777928    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-kubernetes\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778187    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778329    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-modprobe-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778367    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778416    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-hostroot\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778503    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-script-lib\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778558    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778598    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf7f273-a82a-40a5-811c-efb7281d1db6-konnectivity-ca\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778560    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778680    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-dbus\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778693    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-multus-certs\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778843    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778883    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-netns\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778915    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-cnibin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778982    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.778984    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-ovn\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779042    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-systemd\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.779866 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779072    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-bin\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779082    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f643dab0-17a4-410b-b56b-f278862fa237-host-slash\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779152    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-kubelet\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779409    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96cf9d61-5fb4-45ed-b68b-868a6738f12c-serviceca\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779554    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779666    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f643dab0-17a4-410b-b56b-f278862fa237-iptables-alerter-script\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779726    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cf9d61-5fb4-45ed-b68b-868a6738f12c-host\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779762    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779774    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-cni-binary-copy\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779798    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-etc-kubernetes\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779903    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-daemon-config\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.779917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-socket-dir-parent\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.780092    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-var-lib-kubelet\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.780211    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-netns\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.780942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.780432    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-config\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.781880 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.781552    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.782508 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.782317    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:12:32.783323 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.783144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.784746 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.784726    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-tmp\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.784827 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.784768    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-tuned\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.784938 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.784920    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf7f273-a82a-40a5-811c-efb7281d1db6-agent-certs\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:32.785415 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.785396    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.796738 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.796712    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prn7r\" (UniqueName: \"kubernetes.io/projected/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-kube-api-access-prn7r\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:32.797920 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.797884    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94g2q\" (UniqueName: \"kubernetes.io/projected/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kube-api-access-94g2q\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.798126 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.798106    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcz9\" (UniqueName: \"kubernetes.io/projected/89e02182-3b19-41c8-9040-f75a4254187b-kube-api-access-kgcz9\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:32.803210 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.803185    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 23:12:32.803303 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.803217    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 23:12:32.803303 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.803233    2582 projected.go:196] Error preparing data for projected volume kube-api-access-26l5f for pod openshift-network-diagnostics/network-check-target-b97hf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:32.803440 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:32.803301    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f podName:534d0681-f061-4879-9e5e-f734eebfc7a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:33.303283228 +0000 UTC m=+2.107308514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-26l5f" (UniqueName: "kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f") pod "network-check-target-b97hf" (UID: "534d0681-f061-4879-9e5e-f734eebfc7a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:32.804268 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.804247    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxp2\" (UniqueName: \"kubernetes.io/projected/d9a2e31a-049d-48d2-979c-8557f7c239f2-kube-api-access-dfxp2\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:32.804963 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.804927    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqjq\" (UniqueName: \"kubernetes.io/projected/96cf9d61-5fb4-45ed-b68b-868a6738f12c-kube-api-access-nzqjq\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.805053 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.805013    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpflj\" (UniqueName: \"kubernetes.io/projected/f643dab0-17a4-410b-b56b-f278862fa237-kube-api-access-lpflj\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:32.805178 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.805156    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fd66\" (UniqueName: \"kubernetes.io/projected/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-kube-api-access-8fd66\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:32.805718 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.805700    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbtb\" (UniqueName: \"kubernetes.io/projected/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-kube-api-access-rgbtb\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:32.806681 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.806662    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqr78\" (UniqueName: \"kubernetes.io/projected/2e327984-d0fc-4330-a933-f104718bce7c-kube-api-access-tqr78\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:32.974761 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.974676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:12:32.983798 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:32.983778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:12:32.985440 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:32.985414    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a24e0c9_e7a1_4206_bdb5_a956b226a79e.slice/crio-eac0e60ef6423aae5e7665e0b612dacf8bda3814d966d1b3d030c7852e9fd241 WatchSource:0}: Error finding container eac0e60ef6423aae5e7665e0b612dacf8bda3814d966d1b3d030c7852e9fd241: Status 404 returned error can't find the container with id eac0e60ef6423aae5e7665e0b612dacf8bda3814d966d1b3d030c7852e9fd241
Oct 29 23:12:32.993321 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:32.993297    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cf9d61_5fb4_45ed_b68b_868a6738f12c.slice/crio-6f205fc76786280c8e65bfa2041c42913ac553f359e25fd0039b651f5e6a292c WatchSource:0}: Error finding container 6f205fc76786280c8e65bfa2041c42913ac553f359e25fd0039b651f5e6a292c: Status 404 returned error can't find the container with id 6f205fc76786280c8e65bfa2041c42913ac553f359e25fd0039b651f5e6a292c
Oct 29 23:12:33.019090 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.019070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:33.020117 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.020100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:12:33.030962 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:33.030022    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b1ae7e_9d23_46d1_ada5_a0e6b66c1418.slice/crio-e3757c3a68734d6ffc6ca1cf6f366e5a6f2b1bfb85d352789488d19bfb18ee2a WatchSource:0}: Error finding container e3757c3a68734d6ffc6ca1cf6f366e5a6f2b1bfb85d352789488d19bfb18ee2a: Status 404 returned error can't find the container with id e3757c3a68734d6ffc6ca1cf6f366e5a6f2b1bfb85d352789488d19bfb18ee2a
Oct 29 23:12:33.033670 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:33.033647    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28ec3c2_c0d3_4312_ad94_16b6b8ba6067.slice/crio-77105b32ebb5c782fe3882300aace9bdbf5d612869e14949c5dde91aca5ba466 WatchSource:0}: Error finding container 77105b32ebb5c782fe3882300aace9bdbf5d612869e14949c5dde91aca5ba466: Status 404 returned error can't find the container with id 77105b32ebb5c782fe3882300aace9bdbf5d612869e14949c5dde91aca5ba466
Oct 29 23:12:33.047054 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.047030    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:12:33.050684 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.050665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:12:33.057591 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:33.057567    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e327984_d0fc_4330_a933_f104718bce7c.slice/crio-e0728485e480488df5fc6c28d939f74a0e94198481fede9f0e4d51038954d6ce WatchSource:0}: Error finding container e0728485e480488df5fc6c28d939f74a0e94198481fede9f0e4d51038954d6ce: Status 404 returned error can't find the container with id e0728485e480488df5fc6c28d939f74a0e94198481fede9f0e4d51038954d6ce
Oct 29 23:12:33.058343 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.058325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:33.059767 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:33.059746    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf643dab0_17a4_410b_b56b_f278862fa237.slice/crio-5dc065a6af4c6476a6fe945a192375793bec2252fe5613ed0273994d4d0a775a WatchSource:0}: Error finding container 5dc065a6af4c6476a6fe945a192375793bec2252fe5613ed0273994d4d0a775a: Status 404 returned error can't find the container with id 5dc065a6af4c6476a6fe945a192375793bec2252fe5613ed0273994d4d0a775a
Oct 29 23:12:33.068149 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:33.068127    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf7f273_a82a_40a5_811c_efb7281d1db6.slice/crio-e17457179617ce890e6b15d1779d84da3d51cf3f190d3b53a92b535ea17f9317 WatchSource:0}: Error finding container e17457179617ce890e6b15d1779d84da3d51cf3f190d3b53a92b535ea17f9317: Status 404 returned error can't find the container with id e17457179617ce890e6b15d1779d84da3d51cf3f190d3b53a92b535ea17f9317
Oct 29 23:12:33.084052 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.084025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kg5q"
Oct 29 23:12:33.089040 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.089022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:12:33.093589 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:33.093564    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a2e31a_049d_48d2_979c_8557f7c239f2.slice/crio-03c425d6a4884fb9ef45474864d654a9e76987884b5d7bfa8928d7b05b1829a6 WatchSource:0}: Error finding container 03c425d6a4884fb9ef45474864d654a9e76987884b5d7bfa8928d7b05b1829a6: Status 404 returned error can't find the container with id 03c425d6a4884fb9ef45474864d654a9e76987884b5d7bfa8928d7b05b1829a6
Oct 29 23:12:33.098741 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:12:33.098719    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e02182_3b19_41c8_9040_f75a4254187b.slice/crio-863826145f66d91883bb1609c0c8a89a873b6f661d014dd8dc326d87cea21c3e WatchSource:0}: Error finding container 863826145f66d91883bb1609c0c8a89a873b6f661d014dd8dc326d87cea21c3e: Status 404 returned error can't find the container with id 863826145f66d91883bb1609c0c8a89a873b6f661d014dd8dc326d87cea21c3e
Oct 29 23:12:33.280013 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.279909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:33.280013 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.279978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:33.280225 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.280161    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:33.280287 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.280230    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret podName:d9192aec-64e7-4d0f-8066-e236dc75b87d nodeName:}" failed. No retries permitted until 2026-10-29 23:12:34.280210906 +0000 UTC m=+3.084236193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret") pod "global-pull-secret-syncer-mkj2j" (UID: "d9192aec-64e7-4d0f-8066-e236dc75b87d") : object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:33.280287 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.280230    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:33.280393 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.280300    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs podName:1d0b2a03-5d09-4c44-bec0-ec66bdea6781 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:34.280280339 +0000 UTC m=+3.084305636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs") pod "network-metrics-daemon-bc5z8" (UID: "1d0b2a03-5d09-4c44-bec0-ec66bdea6781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:33.381239 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.381196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:33.381468 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.381447    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 23:12:33.381468 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.381468    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 23:12:33.381590 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.381481    2582 projected.go:196] Error preparing data for projected volume kube-api-access-26l5f for pod openshift-network-diagnostics/network-check-target-b97hf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:33.381590 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:33.381560    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f podName:534d0681-f061-4879-9e5e-f734eebfc7a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:34.381534889 +0000 UTC m=+3.185560175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-26l5f" (UniqueName: "kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f") pod "network-check-target-b97hf" (UID: "534d0681-f061-4879-9e5e-f734eebfc7a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:33.574976 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.574850    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:12:33.679616 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.679544    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:07:33 +0000 UTC" deadline="2028-05-18 18:40:24.764832311 +0000 UTC"
Oct 29 23:12:33.679616 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.679614    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13603h27m51.085222347s"
Oct 29 23:12:33.769635 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.769520    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44xj8" event={"ID":"faf7f273-a82a-40a5-811c-efb7281d1db6","Type":"ContainerStarted","Data":"e17457179617ce890e6b15d1779d84da3d51cf3f190d3b53a92b535ea17f9317"}
Oct 29 23:12:33.774641 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.774600    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-glknz" event={"ID":"f643dab0-17a4-410b-b56b-f278862fa237","Type":"ContainerStarted","Data":"5dc065a6af4c6476a6fe945a192375793bec2252fe5613ed0273994d4d0a775a"}
Oct 29 23:12:33.792011 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.791973    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bvzv" event={"ID":"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067","Type":"ContainerStarted","Data":"77105b32ebb5c782fe3882300aace9bdbf5d612869e14949c5dde91aca5ba466"}
Oct 29 23:12:33.801476 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.801439    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwc9k" event={"ID":"96cf9d61-5fb4-45ed-b68b-868a6738f12c","Type":"ContainerStarted","Data":"6f205fc76786280c8e65bfa2041c42913ac553f359e25fd0039b651f5e6a292c"}
Oct 29 23:12:33.807988 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.807956    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn" event={"ID":"2a24e0c9-e7a1-4206-bdb5-a956b226a79e","Type":"ContainerStarted","Data":"eac0e60ef6423aae5e7665e0b612dacf8bda3814d966d1b3d030c7852e9fd241"}
Oct 29 23:12:33.822184 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.822110    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal" event={"ID":"410469b0f63ac5a751ec52b1c6fe6043","Type":"ContainerStarted","Data":"80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696"}
Oct 29 23:12:33.825031 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.824966    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zx5vd" event={"ID":"89e02182-3b19-41c8-9040-f75a4254187b","Type":"ContainerStarted","Data":"863826145f66d91883bb1609c0c8a89a873b6f661d014dd8dc326d87cea21c3e"}
Oct 29 23:12:33.831165 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.831137    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kg5q" event={"ID":"d9a2e31a-049d-48d2-979c-8557f7c239f2","Type":"ContainerStarted","Data":"03c425d6a4884fb9ef45474864d654a9e76987884b5d7bfa8928d7b05b1829a6"}
Oct 29 23:12:33.840686 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.840657    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerStarted","Data":"e0728485e480488df5fc6c28d939f74a0e94198481fede9f0e4d51038954d6ce"}
Oct 29 23:12:33.861809 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.861774    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"e3757c3a68734d6ffc6ca1cf6f366e5a6f2b1bfb85d352789488d19bfb18ee2a"}
Oct 29 23:12:33.865449 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.865420    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal" event={"ID":"88f04efcc82b4a40de30b52c3590d78c","Type":"ContainerStarted","Data":"3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e"}
Oct 29 23:12:33.924341 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:33.924179    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:12:34.289356 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.289273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:34.289356 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.289321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:34.289588 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.289523    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:34.289641 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.289589    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret podName:d9192aec-64e7-4d0f-8066-e236dc75b87d nodeName:}" failed. No retries permitted until 2026-10-29 23:12:36.289568053 +0000 UTC m=+5.093593348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret") pod "global-pull-secret-syncer-mkj2j" (UID: "d9192aec-64e7-4d0f-8066-e236dc75b87d") : object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:34.290025 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.290000    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:34.290160 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.290068    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs podName:1d0b2a03-5d09-4c44-bec0-ec66bdea6781 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:36.290051845 +0000 UTC m=+5.094077142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs") pod "network-metrics-daemon-bc5z8" (UID: "1d0b2a03-5d09-4c44-bec0-ec66bdea6781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:34.390680 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.390648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:34.390868 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.390825    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 23:12:34.390868 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.390845    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 23:12:34.390868 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.390858    2582 projected.go:196] Error preparing data for projected volume kube-api-access-26l5f for pod openshift-network-diagnostics/network-check-target-b97hf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:34.391059 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.390929    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f podName:534d0681-f061-4879-9e5e-f734eebfc7a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:36.39089512 +0000 UTC m=+5.194920417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-26l5f" (UniqueName: "kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f") pod "network-check-target-b97hf" (UID: "534d0681-f061-4879-9e5e-f734eebfc7a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:34.680406 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.680310    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:07:33 +0000 UTC" deadline="2028-05-02 09:22:41.252892729 +0000 UTC"
Oct 29 23:12:34.680406 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.680357    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13210h10m6.572541079s"
Oct 29 23:12:34.759724 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.759686    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:34.759898 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.759829    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:34.760350 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.760270    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:34.760448 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.760370    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:34.760448 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:34.760444    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:34.760557 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:34.760527    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:35.235417 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:35.235381    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:12:36.307543 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:36.307510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:36.308044 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:36.307591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:36.308044 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.307688    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:36.308044 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.307710    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:36.308044 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.307769    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs podName:1d0b2a03-5d09-4c44-bec0-ec66bdea6781 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:40.307755877 +0000 UTC m=+9.111781172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs") pod "network-metrics-daemon-bc5z8" (UID: "1d0b2a03-5d09-4c44-bec0-ec66bdea6781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:36.308044 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.307806    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret podName:d9192aec-64e7-4d0f-8066-e236dc75b87d nodeName:}" failed. No retries permitted until 2026-10-29 23:12:40.307776205 +0000 UTC m=+9.111801487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret") pod "global-pull-secret-syncer-mkj2j" (UID: "d9192aec-64e7-4d0f-8066-e236dc75b87d") : object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:36.408431 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:36.408387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:36.408617 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.408594    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 23:12:36.408677 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.408624    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 23:12:36.408677 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.408639    2582 projected.go:196] Error preparing data for projected volume kube-api-access-26l5f for pod openshift-network-diagnostics/network-check-target-b97hf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:36.408794 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.408703    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f podName:534d0681-f061-4879-9e5e-f734eebfc7a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:40.408683281 +0000 UTC m=+9.212708567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-26l5f" (UniqueName: "kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f") pod "network-check-target-b97hf" (UID: "534d0681-f061-4879-9e5e-f734eebfc7a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:36.759734 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:36.759699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:36.759942 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.759844    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:36.763047 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:36.760261    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:36.763047 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.760383    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:36.763047 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:36.760429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:36.763047 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:36.760494    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:37.897576 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:37.897316    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="63254f042f1734236d442580f7475c7a9634f92520b660f4a9e7ada0d620f143" exitCode=0
Oct 29 23:12:37.898186 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:37.897478    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerDied","Data":"63254f042f1734236d442580f7475c7a9634f92520b660f4a9e7ada0d620f143"}
Oct 29 23:12:37.905445 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:37.905413    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal" event={"ID":"88f04efcc82b4a40de30b52c3590d78c","Type":"ContainerStarted","Data":"30d989fe00db8aa2d6d8f757f952a13a5501247158e5caeab16a05dae65f6483"}
Oct 29 23:12:37.908612 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:37.908585    2582 generic.go:356] "Generic (PLEG): container finished" podID="410469b0f63ac5a751ec52b1c6fe6043" containerID="04b1d4d4e30774d3a14a505b09b87f0bda92441f0049a381ead4362ef389f5f4" exitCode=0
Oct 29 23:12:37.908677 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:37.908655    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal" event={"ID":"410469b0f63ac5a751ec52b1c6fe6043","Type":"ContainerDied","Data":"04b1d4d4e30774d3a14a505b09b87f0bda92441f0049a381ead4362ef389f5f4"}
Oct 29 23:12:38.027304 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:38.026570    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal" podStartSLOduration=5.02654814 podStartE2EDuration="5.02654814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:38.026195911 +0000 UTC m=+6.830221244" watchObservedRunningTime="2026-10-29 23:12:38.02654814 +0000 UTC m=+6.830573446"
Oct 29 23:12:38.759689 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:38.759653    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:38.759892 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:38.759809    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:38.760270 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:38.760249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:38.760395 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:38.760356    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:38.760463 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:38.760417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:38.760513 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:38.760501    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:38.916577 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:38.916078    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44xj8" event={"ID":"faf7f273-a82a-40a5-811c-efb7281d1db6","Type":"ContainerStarted","Data":"950009a8391436426a035923f97b2f77bf21309005fffd0c47444ebd5846627c"}
Oct 29 23:12:38.920222 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:38.920188    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal" event={"ID":"410469b0f63ac5a751ec52b1c6fe6043","Type":"ContainerStarted","Data":"b475dfa91cb8980648c73f2e9aa5c8c9f888d86e3556a942ab30f92662451ff6"}
Oct 29 23:12:39.017987 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.017477    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-44xj8" podStartSLOduration=2.349044691 podStartE2EDuration="7.017462513s" totalImagesPullingTime="4.668417822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:38.963185268 +0000 UTC m=+7.767210574" watchObservedRunningTime="2026-10-29 23:12:39.017462513 +0000 UTC m=+7.821487817"
Oct 29 23:12:39.022250 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.021122    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal" podStartSLOduration=6.021107629 podStartE2EDuration="6.021107629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:39.017151178 +0000 UTC m=+7.821176484" watchObservedRunningTime="2026-10-29 23:12:39.021107629 +0000 UTC m=+7.825132935"
Oct 29 23:12:39.190554 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.190280    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:39.190970 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.190944    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:39.926366 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.925518    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwc9k" event={"ID":"96cf9d61-5fb4-45ed-b68b-868a6738f12c","Type":"ContainerStarted","Data":"8538e1f91a0c1e98749f4900f7b4963f6f2332b3e298e1b0941336fecbc516fd"}
Oct 29 23:12:39.931550 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.930869    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn" event={"ID":"2a24e0c9-e7a1-4206-bdb5-a956b226a79e","Type":"ContainerStarted","Data":"609ef964718d47ce2e125f1bf902ccfe9d6d3a5c5de23955685f42b0360fc92d"}
Oct 29 23:12:39.931550 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.930934    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:39.932344 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.932285    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:12:39.971319 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:39.971253    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qwc9k" podStartSLOduration=2.027486156 podStartE2EDuration="7.971231964s" totalImagesPullingTime="5.943745808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:39.970561802 +0000 UTC m=+8.774587108" watchObservedRunningTime="2026-10-29 23:12:39.971231964 +0000 UTC m=+8.775257270"
Oct 29 23:12:40.346016 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:40.345909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:40.346016 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:40.345970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:40.346256 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.346118    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:40.346256 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.346192    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret podName:d9192aec-64e7-4d0f-8066-e236dc75b87d nodeName:}" failed. No retries permitted until 2026-10-29 23:12:48.346171856 +0000 UTC m=+17.150197147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret") pod "global-pull-secret-syncer-mkj2j" (UID: "d9192aec-64e7-4d0f-8066-e236dc75b87d") : object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:40.346624 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.346602    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:40.346725 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.346661    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs podName:1d0b2a03-5d09-4c44-bec0-ec66bdea6781 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:48.346645245 +0000 UTC m=+17.150670532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs") pod "network-metrics-daemon-bc5z8" (UID: "1d0b2a03-5d09-4c44-bec0-ec66bdea6781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:40.447200 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:40.447157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:40.447436 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.447414    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 23:12:40.447504 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.447450    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 23:12:40.447504 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.447468    2582 projected.go:196] Error preparing data for projected volume kube-api-access-26l5f for pod openshift-network-diagnostics/network-check-target-b97hf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:40.447617 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.447538    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f podName:534d0681-f061-4879-9e5e-f734eebfc7a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:12:48.447515468 +0000 UTC m=+17.251540793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-26l5f" (UniqueName: "kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f") pod "network-check-target-b97hf" (UID: "534d0681-f061-4879-9e5e-f734eebfc7a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:40.760055 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:40.759425    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:40.760055 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.759546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:40.760055 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:40.759877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:40.760055 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.759976    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:40.760373 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:40.760029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:40.760373 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:40.760174    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:41.389353 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:41.389314    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 23:12:41.641462 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:41.641371    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T23:12:41.389347584Z","UUID":"752a838d-8c6c-406f-9800-613eba94a99a","Handler":null,"Name":"","Endpoint":""}
Oct 29 23:12:41.648101 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:41.648063    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 23:12:41.648294 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:41.648126    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 23:12:41.941823 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:41.940706    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn" event={"ID":"2a24e0c9-e7a1-4206-bdb5-a956b226a79e","Type":"ContainerStarted","Data":"5449313d7a7fb2712e6c21fc51cac438bf53fd339e94f0d32ae2b8a771f1a3d0"}
Oct 29 23:12:41.944253 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:41.944206    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kg5q" event={"ID":"d9a2e31a-049d-48d2-979c-8557f7c239f2","Type":"ContainerStarted","Data":"8f5d46325c2222c4b3b34df54cd66868926eb3f46b8f2511739d11b9df61d409"}
Oct 29 23:12:42.759383 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:42.759350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:42.759829 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:42.759408    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:42.759829 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:42.759509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:42.759829 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:42.759527    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:42.759829 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:42.759607    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:42.759829 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:42.759681    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:42.948816 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:42.948744    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn" event={"ID":"2a24e0c9-e7a1-4206-bdb5-a956b226a79e","Type":"ContainerStarted","Data":"ddb30ede384d0b6df902cce47e9cc1eed4474d72117ae06a45c09be4edae3fa2"}
Oct 29 23:12:42.950399 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:42.950364    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zx5vd" event={"ID":"89e02182-3b19-41c8-9040-f75a4254187b","Type":"ContainerStarted","Data":"5f7d64c43f17949eb15cb6096298d027dddc4a1dc11d0d1b359b1800a9cf3ef9"}
Oct 29 23:12:42.956325 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:42.955247    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="d9abae1b4a08887c3f435fc97992e624e8a763b7187277243d6bf87a9f1ae623" exitCode=0
Oct 29 23:12:42.956325 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:42.955356    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerDied","Data":"d9abae1b4a08887c3f435fc97992e624e8a763b7187277243d6bf87a9f1ae623"}
Oct 29 23:12:43.005306 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:43.005247    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2kg5q" podStartSLOduration=2.584649784 podStartE2EDuration="11.005230645s" totalImagesPullingTime="8.420580861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:41.996685909 +0000 UTC m=+10.800711226" watchObservedRunningTime="2026-10-29 23:12:43.005230645 +0000 UTC m=+11.809255944"
Oct 29 23:12:43.005799 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:43.005769    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn" podStartSLOduration=1.979492668 podStartE2EDuration="11.005760551s" totalImagesPullingTime="9.026267883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:43.005219087 +0000 UTC m=+11.809244388" watchObservedRunningTime="2026-10-29 23:12:43.005760551 +0000 UTC m=+11.809785851"
Oct 29 23:12:43.047704 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:43.047648    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zx5vd" podStartSLOduration=2.301788272 podStartE2EDuration="11.047635331s" totalImagesPullingTime="8.745847059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:43.047183634 +0000 UTC m=+11.851208935" watchObservedRunningTime="2026-10-29 23:12:43.047635331 +0000 UTC m=+11.851660633"
Oct 29 23:12:43.960725 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:43.960635    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="a292bc58cf2bd00353c5b034c73feb69866c20ba9a0d0dc574bc58e8459b4b7e" exitCode=0
Oct 29 23:12:43.961249 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:43.960726    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerDied","Data":"a292bc58cf2bd00353c5b034c73feb69866c20ba9a0d0dc574bc58e8459b4b7e"}
Oct 29 23:12:43.962563 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:43.962534    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-glknz" event={"ID":"f643dab0-17a4-410b-b56b-f278862fa237","Type":"ContainerStarted","Data":"5c7ff155468e5162c2d1e152b7e5250259321f377baab53b35670cf5cc804cb2"}
Oct 29 23:12:44.057587 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:44.057530    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-glknz" podStartSLOduration=3.00850136 podStartE2EDuration="12.057515721s" totalImagesPullingTime="9.049014361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:44.057268734 +0000 UTC m=+12.861294035" watchObservedRunningTime="2026-10-29 23:12:44.057515721 +0000 UTC m=+12.861541021"
Oct 29 23:12:44.759498 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:44.759419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:44.759650 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:44.759513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:44.759650 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:44.759620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:44.759757 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:44.759640    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:44.759757 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:44.759737    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:44.759861 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:44.759801    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:44.966115 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:44.966076    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4bvzv" event={"ID":"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067","Type":"ContainerStarted","Data":"c6e1dcf7afc98d931fc8128dbce8f1e43cfe9fda48e6b09c19e7ce39f67e3d30"}
Oct 29 23:12:44.968871 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:44.968839    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="76ef1da2368ec878f99f9d42047532f738cb480daf4bd905e7762a32c679818b" exitCode=0
Oct 29 23:12:44.968997 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:44.968911    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerDied","Data":"76ef1da2368ec878f99f9d42047532f738cb480daf4bd905e7762a32c679818b"}
Oct 29 23:12:45.012725 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:45.012626    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4bvzv" podStartSLOduration=1.608680604 podStartE2EDuration="13.012606953s" totalImagesPullingTime="11.403926349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:45.011596856 +0000 UTC m=+13.815622157" watchObservedRunningTime="2026-10-29 23:12:45.012606953 +0000 UTC m=+13.816632253"
Oct 29 23:12:46.759207 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:46.759175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:46.759207 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:46.759199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:46.759731 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:46.759288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:46.759731 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:46.759302    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:46.759731 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:46.759368    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:46.759731 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:46.759457    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:46.976960 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:46.976626    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="92e5f0680859cfdc12a2f1e02017b6a90b37babedfcc024813051c21e59547ce" exitCode=0
Oct 29 23:12:46.976960 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:46.976716    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerDied","Data":"92e5f0680859cfdc12a2f1e02017b6a90b37babedfcc024813051c21e59547ce"}
Oct 29 23:12:47.984864 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:47.982401    2582 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="adecc820fc452016731415d37d6bcafab7fa51688a3de2fa7487ea9e89cdfb0f" exitCode=0
Oct 29 23:12:47.984864 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:47.982465    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerDied","Data":"adecc820fc452016731415d37d6bcafab7fa51688a3de2fa7487ea9e89cdfb0f"}
Oct 29 23:12:48.415562 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.415478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:48.415562 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.415516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:48.415741 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.415646    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:48.415741 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.415685    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:48.415741 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.415719    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs podName:1d0b2a03-5d09-4c44-bec0-ec66bdea6781 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:04.415698782 +0000 UTC m=+33.219724074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs") pod "network-metrics-daemon-bc5z8" (UID: "1d0b2a03-5d09-4c44-bec0-ec66bdea6781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:12:48.415741 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.415735    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret podName:d9192aec-64e7-4d0f-8066-e236dc75b87d nodeName:}" failed. No retries permitted until 2026-10-29 23:13:04.415728719 +0000 UTC m=+33.219753998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret") pod "global-pull-secret-syncer-mkj2j" (UID: "d9192aec-64e7-4d0f-8066-e236dc75b87d") : object "kube-system"/"original-pull-secret" not registered
Oct 29 23:12:48.516733 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.516695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:48.516905 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.516838    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 23:12:48.516905 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.516856    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 23:12:48.516905 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.516867    2582 projected.go:196] Error preparing data for projected volume kube-api-access-26l5f for pod openshift-network-diagnostics/network-check-target-b97hf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:48.517007 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.516935    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f podName:534d0681-f061-4879-9e5e-f734eebfc7a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:04.51691734 +0000 UTC m=+33.320942644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-26l5f" (UniqueName: "kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f") pod "network-check-target-b97hf" (UID: "534d0681-f061-4879-9e5e-f734eebfc7a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:12:48.759937 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.759913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:48.760032 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.759911    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:48.760032 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.760015    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:48.760154 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.759924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:48.760154 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.760076    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:48.760254 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:48.760177    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:48.988095 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.987785    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzt69" event={"ID":"2e327984-d0fc-4330-a933-f104718bce7c","Type":"ContainerStarted","Data":"2cdc15d267e00b073abf21a64e5e94c3585001ce0a141b9040f8b202063f470f"}
Oct 29 23:12:48.990680 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.990647    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"e59569f74ffe2bddb0b26302538afee9cc04589bb780cd23e44e40eaa6bc4910"}
Oct 29 23:12:48.990769 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.990716    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"8e8618d388bb33455bc9412d86b72d6b19aba77d9eae20c6191a1ca1409ab513"}
Oct 29 23:12:48.990769 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.990749    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"b713dac96970ec236b741112733e2a3c9dbfedde6203381d152691633694cb2f"}
Oct 29 23:12:48.990841 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.990777    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"fdce09a0705c2f26abb6da372e6457afc1897e99105d8f7f43934788a42cbe39"}
Oct 29 23:12:48.990841 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.990807    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"c9ed6e391066d110344d1edbb4059e9b242d091d281d558d8fb3dbabaf27ab8a"}
Oct 29 23:12:48.990841 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:48.990834    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"3c9854c958a5f55f243bd43973a44d9d28889acf38c44ce341ef2322e2e2e4f4"}
Oct 29 23:12:49.040717 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:49.040673    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mzt69" podStartSLOduration=6.121099957 podStartE2EDuration="17.040662503s" totalImagesPullingTime="10.919562546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:49.040442541 +0000 UTC m=+17.844467841" watchObservedRunningTime="2026-10-29 23:12:49.040662503 +0000 UTC m=+17.844687804"
Oct 29 23:12:50.404989 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:50.404961    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zx5vd_89e02182-3b19-41c8-9040-f75a4254187b/dns-node-resolver/0.log"
Oct 29 23:12:50.759941 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:50.759837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:50.759941 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:50.759860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:50.760169 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:50.759988    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:50.760169 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:50.760031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:50.760169 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:50.760113    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:50.760295 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:50.760191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:50.998587 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:50.998553    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"5fba62d5b21a5a3bcc70a0163b1bdffdbc8c848a74882a039495fac27140dd67"}
Oct 29 23:12:51.139831 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:51.139755    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwc9k_96cf9d61-5fb4-45ed-b68b-868a6738f12c/node-ca/0.log"
Oct 29 23:12:52.759708 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:52.759641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:52.759708 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:52.759724    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:52.760275 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:52.759836    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:52.760275 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:52.759893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:52.760275 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:52.759961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:52.760275 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:52.760048    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:53.259347 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:53.259315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44xj8_faf7f273-a82a-40a5-811c-efb7281d1db6/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:53.441596 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:53.441519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44xj8_faf7f273-a82a-40a5-811c-efb7281d1db6/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:53.614356 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:53.614321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44xj8_faf7f273-a82a-40a5-811c-efb7281d1db6/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:53.781085 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:53.781010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44xj8_faf7f273-a82a-40a5-811c-efb7281d1db6/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:54.009674 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.009462    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" event={"ID":"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418","Type":"ContainerStarted","Data":"c3b3a845569e55a96cd03d098e3d1e2e1b4ffee0712fd19d3d1bddf17bec5a62"}
Oct 29 23:12:54.009851 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.009728    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:54.009851 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.009782    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:54.025138 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.025117    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:54.103356 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.103266    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws" podStartSLOduration=6.610667257 podStartE2EDuration="22.103241295s" totalImagesPullingTime="15.492574038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:54.089821761 +0000 UTC m=+22.893847106" watchObservedRunningTime="2026-10-29 23:12:54.103241295 +0000 UTC m=+22.907266593"
Oct 29 23:12:54.758831 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.758802    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:54.758831 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.758824    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:54.759093 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:54.758923    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:54.759093 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:54.758985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:54.759093 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:54.759068    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:54.759246 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:54.759100    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:55.011571 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:55.011482    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:55.026568 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:55.026545    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:12:56.759103 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:56.759068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:56.759550 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:56.759141    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:56.759550 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:56.759234    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:56.759550 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:56.759289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:56.759550 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:56.759369    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:12:56.759550 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:56.759439    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:58.758932 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:58.758895    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:12:58.759397 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:58.758895    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:12:58.759397 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:58.759010    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:12:58.759397 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:58.759069    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:12:58.759397 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:12:58.758895    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:12:58.759397 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:12:58.759137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:00.759142 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:00.759109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:00.759527 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:00.759164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:00.759527 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:00.759251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:00.759527 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:00.759249    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:00.759527 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:00.759313    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:00.759527 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:00.759381    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:02.759267 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:02.759226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:02.759720 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:02.759275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:02.759720 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:02.759287    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:02.759720 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:02.759362    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:02.759720 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:02.759513    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:02.759720 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:02.759596    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:04.424017 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:04.423987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:04.424459 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:04.424049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:04.424459 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.424137    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:13:04.424459 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.424139    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 23:13:04.424459 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.424184    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs podName:1d0b2a03-5d09-4c44-bec0-ec66bdea6781 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:36.424171364 +0000 UTC m=+65.228196644 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs") pod "network-metrics-daemon-bc5z8" (UID: "1d0b2a03-5d09-4c44-bec0-ec66bdea6781") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 23:13:04.424459 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.424196    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret podName:d9192aec-64e7-4d0f-8066-e236dc75b87d nodeName:}" failed. No retries permitted until 2026-10-29 23:13:36.42419012 +0000 UTC m=+65.228215398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret") pod "global-pull-secret-syncer-mkj2j" (UID: "d9192aec-64e7-4d0f-8066-e236dc75b87d") : object "kube-system"/"original-pull-secret" not registered
Oct 29 23:13:04.524687 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:04.524648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:04.524855 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.524803    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 23:13:04.524855 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.524822    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 23:13:04.524855 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.524831    2582 projected.go:196] Error preparing data for projected volume kube-api-access-26l5f for pod openshift-network-diagnostics/network-check-target-b97hf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:13:04.524972 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.524892    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f podName:534d0681-f061-4879-9e5e-f734eebfc7a4 nodeName:}" failed. No retries permitted until 2026-10-29 23:13:36.524866191 +0000 UTC m=+65.328891471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-26l5f" (UniqueName: "kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f") pod "network-check-target-b97hf" (UID: "534d0681-f061-4879-9e5e-f734eebfc7a4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 23:13:04.759634 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:04.759547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:04.759634 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:04.759582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:04.759634 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:04.759582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:04.759828 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.759658    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:04.759828 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.759745    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:04.759828 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:04.759794    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:06.759209 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:06.759180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:06.759589 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:06.759250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:06.759589 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:06.759275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:06.759589 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:06.759351    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:06.759589 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:06.759390    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:06.759589 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:06.759460    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:08.759442 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:08.759398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:08.759828 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:08.759449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:08.759828 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:08.759419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:08.759828 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:08.759533    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:08.759828 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:08.759576    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:08.759828 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:08.759660    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:10.158581 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:10.158536    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mkj2j"]
Oct 29 23:13:10.159084 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:10.158689    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:10.159084 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:10.158798    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:10.159233 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:10.159210    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b97hf"]
Oct 29 23:13:10.159601 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:10.159342    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:10.159601 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:10.159437    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:10.161745 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:10.161722    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bc5z8"]
Oct 29 23:13:10.161902 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:10.161867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:10.162025 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:10.161991    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:11.389510 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:11.389440    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zx5vd_89e02182-3b19-41c8-9040-f75a4254187b/dns-node-resolver/0.log"
Oct 29 23:13:11.762222 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:11.762043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:11.762410 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:11.762094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:11.762410 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:11.762299    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:11.762410 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:11.762094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:11.762410 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:11.762353    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:11.762410 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:11.762409    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:12.388528 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:12.388501    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwc9k_96cf9d61-5fb4-45ed-b68b-868a6738f12c/node-ca/0.log"
Oct 29 23:13:13.759092 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:13.759061    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:13.759092 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:13.759079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:13.759534 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:13.759191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b97hf" podUID="534d0681-f061-4879-9e5e-f734eebfc7a4"
Oct 29 23:13:13.759534 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:13.759232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:13.759534 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:13.759332    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mkj2j" podUID="d9192aec-64e7-4d0f-8066-e236dc75b87d"
Oct 29 23:13:13.759534 ip-10-0-139-224 kubenswrapper[2582]: E1029 23:13:13.759398    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bc5z8" podUID="1d0b2a03-5d09-4c44-bec0-ec66bdea6781"
Oct 29 23:13:15.465633 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.465601    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeReady"
Oct 29 23:13:15.466042 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.465722    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 23:13:15.563760 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.563734    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gkt7z"]
Oct 29 23:13:15.565847 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.565832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.583638 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.583617    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mfwdq"]
Oct 29 23:13:15.585437 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.585424    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.594725 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.594702    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gkt7z"]
Oct 29 23:13:15.608362 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.608337    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfwdq"]
Oct 29 23:13:15.645038 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.645008    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kjc8k"]
Oct 29 23:13:15.646915 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.646900    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.653837 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.653816    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 23:13:15.653997 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.653978    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rlkhk\""
Oct 29 23:13:15.653997 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.653990    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 23:13:15.654140 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.653995    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88spp\""
Oct 29 23:13:15.654140 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.654136    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 23:13:15.654201 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.654171    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 23:13:15.654294 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.654282    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 23:13:15.654368 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.654353    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 23:13:15.657850 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.657836    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 23:13:15.672367 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.672343    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kjc8k"]
Oct 29 23:13:15.714064 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.714039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9685c622-c264-4e53-8542-7a07650a6e84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.714204 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.714085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgrh6\" (UniqueName: \"kubernetes.io/projected/9685c622-c264-4e53-8542-7a07650a6e84-kube-api-access-vgrh6\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.714204 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.714114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9685c622-c264-4e53-8542-7a07650a6e84-crio-socket\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.714204 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.714176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtk22\" (UniqueName: \"kubernetes.io/projected/f88a9388-d3c1-421a-bb58-87a481f7333a-kube-api-access-mtk22\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.714314 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.714273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88a9388-d3c1-421a-bb58-87a481f7333a-cert\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.714314 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.714305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9685c622-c264-4e53-8542-7a07650a6e84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.714383 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.714324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9685c622-c264-4e53-8542-7a07650a6e84-data-volume\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.752753 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.752690    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cxqm7\""
Oct 29 23:13:15.752753 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.752690    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 23:13:15.753380 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.753366    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 23:13:15.759635 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.759622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:15.759683 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.759647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:15.759918 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.759899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:15.814756 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.814729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1769c496-3b59-4aba-a674-13dae505ed09-metrics-tls\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.814910 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.814773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrh6\" (UniqueName: \"kubernetes.io/projected/9685c622-c264-4e53-8542-7a07650a6e84-kube-api-access-vgrh6\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.814910 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.814792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9685c622-c264-4e53-8542-7a07650a6e84-crio-socket\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.814910 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.814866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtk22\" (UniqueName: \"kubernetes.io/projected/f88a9388-d3c1-421a-bb58-87a481f7333a-kube-api-access-mtk22\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.815052 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.814948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88a9388-d3c1-421a-bb58-87a481f7333a-cert\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.815052 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.814985    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7k8\" (UniqueName: \"kubernetes.io/projected/1769c496-3b59-4aba-a674-13dae505ed09-kube-api-access-5k7k8\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.815052 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9685c622-c264-4e53-8542-7a07650a6e84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.815176 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9685c622-c264-4e53-8542-7a07650a6e84-data-volume\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.815176 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9685c622-c264-4e53-8542-7a07650a6e84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.815176 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1769c496-3b59-4aba-a674-13dae505ed09-config-volume\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.815314 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815180    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9685c622-c264-4e53-8542-7a07650a6e84-crio-socket\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.815314 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815202    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1769c496-3b59-4aba-a674-13dae505ed09-tmp-dir\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.815535 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815512    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9685c622-c264-4e53-8542-7a07650a6e84-data-volume\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.815743 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.815723    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9685c622-c264-4e53-8542-7a07650a6e84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.818683 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.818656    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9685c622-c264-4e53-8542-7a07650a6e84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.818683 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.818670    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88a9388-d3c1-421a-bb58-87a481f7333a-cert\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.839855 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.839836    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrh6\" (UniqueName: \"kubernetes.io/projected/9685c622-c264-4e53-8542-7a07650a6e84-kube-api-access-vgrh6\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.840015 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.839994    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtk22\" (UniqueName: \"kubernetes.io/projected/f88a9388-d3c1-421a-bb58-87a481f7333a-kube-api-access-mtk22\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.853568 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.853484    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m6t5c\""
Oct 29 23:13:15.853568 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.853512    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z28mk\""
Oct 29 23:13:15.853568 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.853511    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wscf4\""
Oct 29 23:13:15.853568 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.853532    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 23:13:15.853568 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.853565    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 23:13:15.853790 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.853572    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 23:13:15.853790 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.853617    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 23:13:15.878610 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.878590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:15.894750 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.894728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:15.915871 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.915836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1769c496-3b59-4aba-a674-13dae505ed09-config-volume\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.916068 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.916047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1769c496-3b59-4aba-a674-13dae505ed09-tmp-dir\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.916166 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.916107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1769c496-3b59-4aba-a674-13dae505ed09-metrics-tls\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.916381 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.916359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7k8\" (UniqueName: \"kubernetes.io/projected/1769c496-3b59-4aba-a674-13dae505ed09-kube-api-access-5k7k8\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.916974 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.916950    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1769c496-3b59-4aba-a674-13dae505ed09-tmp-dir\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.917146 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.917120    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1769c496-3b59-4aba-a674-13dae505ed09-config-volume\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.919947 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.919609    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1769c496-3b59-4aba-a674-13dae505ed09-metrics-tls\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.950351 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.950320    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7k8\" (UniqueName: \"kubernetes.io/projected/1769c496-3b59-4aba-a674-13dae505ed09-kube-api-access-5k7k8\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:15.957598 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:15.957352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:16.219346 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.219311    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfwdq"]
Oct 29 23:13:16.223240 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:16.223211    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88a9388_d3c1_421a_bb58_87a481f7333a.slice/crio-edd42781e655509d8bf4b48fee36140e35717a2b3d3531bca0514643d7a95c1c WatchSource:0}: Error finding container edd42781e655509d8bf4b48fee36140e35717a2b3d3531bca0514643d7a95c1c: Status 404 returned error can't find the container with id edd42781e655509d8bf4b48fee36140e35717a2b3d3531bca0514643d7a95c1c
Oct 29 23:13:16.229790 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.229580    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gkt7z"]
Oct 29 23:13:16.236515 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:16.236488    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9685c622_c264_4e53_8542_7a07650a6e84.slice/crio-e92b562f3610f65280cd7226e8e09a8ad1906335a4ba02f4d2313792519be2af WatchSource:0}: Error finding container e92b562f3610f65280cd7226e8e09a8ad1906335a4ba02f4d2313792519be2af: Status 404 returned error can't find the container with id e92b562f3610f65280cd7226e8e09a8ad1906335a4ba02f4d2313792519be2af
Oct 29 23:13:16.278556 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.278536    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kjc8k"]
Oct 29 23:13:16.281274 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:16.281236    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1769c496_3b59_4aba_a674_13dae505ed09.slice/crio-a259acccbe5603c4abee3e6957f5a1c5af92e311b33c144595f79c10128ae777 WatchSource:0}: Error finding container a259acccbe5603c4abee3e6957f5a1c5af92e311b33c144595f79c10128ae777: Status 404 returned error can't find the container with id a259acccbe5603c4abee3e6957f5a1c5af92e311b33c144595f79c10128ae777
Oct 29 23:13:16.543609 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.543530    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"]
Oct 29 23:13:16.545718 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.545697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.569982 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.569933    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 23:13:16.570125 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.569994    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 23:13:16.570125 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.570095    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 23:13:16.574246 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.574227    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5hpb5\""
Oct 29 23:13:16.579120 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.579091    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 29 23:13:16.581162 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.581128    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 29 23:13:16.585914 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.585459    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"]
Oct 29 23:13:16.598183 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.598153    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"]
Oct 29 23:13:16.600511 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.600490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.610223 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.610202    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f24wt"]
Oct 29 23:13:16.612920 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.612902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.634452 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.634421    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"]
Oct 29 23:13:16.638957 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.638933    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 29 23:13:16.639087 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.638941    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 23:13:16.675769 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.675744    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 23:13:16.675956 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.675905    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 29 23:13:16.676049 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.676035    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8hh8s\""
Oct 29 23:13:16.676144 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.676127    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 29 23:13:16.676269 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.676248    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rxpzs\""
Oct 29 23:13:16.676356 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.676344    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 23:13:16.721916 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.721854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722076 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.721955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.722076 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.721984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722076 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-root\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722076 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-textfile\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722076 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-wtmp\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722076 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw22s\" (UniqueName: \"kubernetes.io/projected/71eca11c-bf4c-473c-b64a-64041bbe808d-kube-api-access-vw22s\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722076 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-metrics-client-ca\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r88bf\" (UniqueName: \"kubernetes.io/projected/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-kube-api-access-r88bf\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-sys\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722161    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722245    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-tls\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.722303 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbshl\" (UniqueName: \"kubernetes.io/projected/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-api-access-fbshl\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.722633 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.722306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.823057 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.822980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-wtmp\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.823057 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw22s\" (UniqueName: \"kubernetes.io/projected/71eca11c-bf4c-473c-b64a-64041bbe808d-kube-api-access-vw22s\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.823256 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-metrics-client-ca\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.823256 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.823256 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88bf\" (UniqueName: \"kubernetes.io/projected/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-kube-api-access-r88bf\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.823256 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-sys\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.823256 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823180    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-wtmp\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.823922 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-metrics-client-ca\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.824039 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823959    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.824271 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.823188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.824393 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.824393 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.824393 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.824393 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-tls\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbshl\" (UniqueName: \"kubernetes.io/projected/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-api-access-fbshl\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-root\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.824701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.824694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-textfile\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.825248 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.825072    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-textfile\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.825248 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.825074    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.825415 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.825397    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-sys\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.825852 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.825832    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.825961 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.825917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.826022 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.825978    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-root\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.830422 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.830393    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.830765 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.830718    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.830858 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.830813    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.831448 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.831221    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.831448 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.831395    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.832777 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.832759    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-tls\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.856582 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.856553    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88bf\" (UniqueName: \"kubernetes.io/projected/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-kube-api-access-r88bf\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.856582 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.856568    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbshl\" (UniqueName: \"kubernetes.io/projected/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-api-access-fbshl\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.857129 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.857106    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw22s\" (UniqueName: \"kubernetes.io/projected/71eca11c-bf4c-473c-b64a-64041bbe808d-kube-api-access-vw22s\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.857653 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.857631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:16.917201 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.916733    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:16.926085 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:16.925897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:16.951770 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:16.951739    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71eca11c_bf4c_473c_b64a_64041bbe808d.slice/crio-cf86a9fc8d17711a7fdd5ad341258cef7fa9bfc867dbfc5cfcf881e419300acc WatchSource:0}: Error finding container cf86a9fc8d17711a7fdd5ad341258cef7fa9bfc867dbfc5cfcf881e419300acc: Status 404 returned error can't find the container with id cf86a9fc8d17711a7fdd5ad341258cef7fa9bfc867dbfc5cfcf881e419300acc
Oct 29 23:13:17.063191 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.063149    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkt7z" event={"ID":"9685c622-c264-4e53-8542-7a07650a6e84","Type":"ContainerStarted","Data":"561926e44086e81334d0ec295021de051d81899f9445605f633c6af9475077b6"}
Oct 29 23:13:17.063381 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.063222    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkt7z" event={"ID":"9685c622-c264-4e53-8542-7a07650a6e84","Type":"ContainerStarted","Data":"83a2e3591e1ae832251c661efd69af129cc7773ee8ea3f8f70a688cde3aa13f9"}
Oct 29 23:13:17.063381 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.063249    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkt7z" event={"ID":"9685c622-c264-4e53-8542-7a07650a6e84","Type":"ContainerStarted","Data":"e92b562f3610f65280cd7226e8e09a8ad1906335a4ba02f4d2313792519be2af"}
Oct 29 23:13:17.064803 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.064751    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f24wt" event={"ID":"71eca11c-bf4c-473c-b64a-64041bbe808d","Type":"ContainerStarted","Data":"cf86a9fc8d17711a7fdd5ad341258cef7fa9bfc867dbfc5cfcf881e419300acc"}
Oct 29 23:13:17.066396 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.066372    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjc8k" event={"ID":"1769c496-3b59-4aba-a674-13dae505ed09","Type":"ContainerStarted","Data":"a259acccbe5603c4abee3e6957f5a1c5af92e311b33c144595f79c10128ae777"}
Oct 29 23:13:17.068313 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.068284    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfwdq" event={"ID":"f88a9388-d3c1-421a-bb58-87a481f7333a","Type":"ContainerStarted","Data":"edd42781e655509d8bf4b48fee36140e35717a2b3d3531bca0514643d7a95c1c"}
Oct 29 23:13:17.185217 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.184542    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"]
Oct 29 23:13:17.188284 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:17.188252    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc1c0f0_73f0_4745_8e51_1d6aa2d14c53.slice/crio-8d1ffca31f6f8bdb3bc7c0fc1f1e02ceda76c8883a16ea07fe4ccbf0c5eff5ec WatchSource:0}: Error finding container 8d1ffca31f6f8bdb3bc7c0fc1f1e02ceda76c8883a16ea07fe4ccbf0c5eff5ec: Status 404 returned error can't find the container with id 8d1ffca31f6f8bdb3bc7c0fc1f1e02ceda76c8883a16ea07fe4ccbf0c5eff5ec
Oct 29 23:13:17.242121 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:17.242089    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"]
Oct 29 23:13:17.244235 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:17.244196    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677ae4b3_e7e3_4b5d_9eaf_d35a0df3b82f.slice/crio-ad9e327416fb0dbce28d069d845d6aaede4b72ffc5b931448ab44b95af4fffd3 WatchSource:0}: Error finding container ad9e327416fb0dbce28d069d845d6aaede4b72ffc5b931448ab44b95af4fffd3: Status 404 returned error can't find the container with id ad9e327416fb0dbce28d069d845d6aaede4b72ffc5b931448ab44b95af4fffd3
Oct 29 23:13:18.075926 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.075604    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m" event={"ID":"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f","Type":"ContainerStarted","Data":"ad9e327416fb0dbce28d069d845d6aaede4b72ffc5b931448ab44b95af4fffd3"}
Oct 29 23:13:18.078507 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.078318    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg" event={"ID":"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53","Type":"ContainerStarted","Data":"d478a4d6ed5839c29ba28e4418665b72a0fed8d625b83619a850283dfd85f06d"}
Oct 29 23:13:18.078507 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.078377    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg" event={"ID":"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53","Type":"ContainerStarted","Data":"318425b12514ed440b5d7b6971d2c85e59b1e60a8ea36d7ed09db3b7fd952498"}
Oct 29 23:13:18.078507 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.078406    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg" event={"ID":"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53","Type":"ContainerStarted","Data":"8d1ffca31f6f8bdb3bc7c0fc1f1e02ceda76c8883a16ea07fe4ccbf0c5eff5ec"}
Oct 29 23:13:18.710446 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.708830    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-866cccd46d-46pmg"]
Oct 29 23:13:18.712630 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.712610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.745703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.745754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.745792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8cb\" (UniqueName: \"kubernetes.io/projected/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-kube-api-access-zm8cb\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.745870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.745950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.745982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-metrics-client-ca\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.746012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-tmp\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.746037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-grpc-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.746301 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.746088    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.748179 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.747936    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3da6io525po6t\""
Oct 29 23:13:18.748179 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.747991    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 29 23:13:18.748179 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.748009    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 29 23:13:18.748179 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.748058    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zlts4\""
Oct 29 23:13:18.748179 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.748120    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 29 23:13:18.748179 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.748137    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 29 23:13:18.748517 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.748272    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 29 23:13:18.752838 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.751089    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-866cccd46d-46pmg"]
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8cb\" (UniqueName: \"kubernetes.io/projected/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-kube-api-access-zm8cb\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-metrics-client-ca\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-tmp\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.848293 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.847397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-grpc-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.849629 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.849216    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-tmp\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.849629 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.849497    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-metrics-client-ca\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.851846 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.851807    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.853754 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.852638    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.853754 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.853102    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.853754 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.853519    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.854553 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.854493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-grpc-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.855138 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.855075    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:18.874485 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:18.874459    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8cb\" (UniqueName: \"kubernetes.io/projected/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-kube-api-access-zm8cb\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:19.068734 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.068703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:19.085095 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.085059    2582 generic.go:356] "Generic (PLEG): container finished" podID="71eca11c-bf4c-473c-b64a-64041bbe808d" containerID="ac55a5c5280e689e670398bba910f9b2192f8599bd36b2148ea69cf935331e37" exitCode=0
Oct 29 23:13:19.085477 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.085195    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f24wt" event={"ID":"71eca11c-bf4c-473c-b64a-64041bbe808d","Type":"ContainerDied","Data":"ac55a5c5280e689e670398bba910f9b2192f8599bd36b2148ea69cf935331e37"}
Oct 29 23:13:19.088105 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.088069    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjc8k" event={"ID":"1769c496-3b59-4aba-a674-13dae505ed09","Type":"ContainerStarted","Data":"08500dba7c49303c9628b4246d61f52e3ded770c97b93e822597111275bef075"}
Oct 29 23:13:19.088206 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.088136    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjc8k" event={"ID":"1769c496-3b59-4aba-a674-13dae505ed09","Type":"ContainerStarted","Data":"cda56f972e65aee7fcbd5cc38d0e36d78a9112120033c160cbd971c3d7ef103e"}
Oct 29 23:13:19.088257 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.088248    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:19.097436 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.095817    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg" event={"ID":"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53","Type":"ContainerStarted","Data":"f955248854242d1bfdb06250d1625054e86823634659afd0cb1a1c458fb862b1"}
Oct 29 23:13:19.183594 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.183056    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg" podStartSLOduration=1.8099308509999998 podStartE2EDuration="3.183038369s" totalImagesPullingTime="1.373107518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:19.182389826 +0000 UTC m=+47.986415129" watchObservedRunningTime="2026-10-29 23:13:19.183038369 +0000 UTC m=+47.987063673"
Oct 29 23:13:19.229587 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.229520    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kjc8k" podStartSLOduration=1.878574187 podStartE2EDuration="4.229504231s" totalImagesPullingTime="2.350930044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:19.228821746 +0000 UTC m=+48.032847077" watchObservedRunningTime="2026-10-29 23:13:19.229504231 +0000 UTC m=+48.033529532"
Oct 29 23:13:19.446449 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:19.446426    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-866cccd46d-46pmg"]
Oct 29 23:13:19.449165 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:19.449133    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e79e4e9_bd92_453f_bb0b_3c3dc213c6f3.slice/crio-fa00c09ca1049acdc8c6b9bed06d4395f5de534e394bdaeb1243763dd551bb8c WatchSource:0}: Error finding container fa00c09ca1049acdc8c6b9bed06d4395f5de534e394bdaeb1243763dd551bb8c: Status 404 returned error can't find the container with id fa00c09ca1049acdc8c6b9bed06d4395f5de534e394bdaeb1243763dd551bb8c
Oct 29 23:13:20.101652 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.101532    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfwdq" event={"ID":"f88a9388-d3c1-421a-bb58-87a481f7333a","Type":"ContainerStarted","Data":"1d69a0844701572f8c5b0d318a75e74bced75c0892736300cdcc09bac20a8e79"}
Oct 29 23:13:20.104038 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.104004    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m" event={"ID":"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f","Type":"ContainerStarted","Data":"de9f5247012636b4dcec0477db7af3b7e02fb50e0b061b10cd9852105316a490"}
Oct 29 23:13:20.104167 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.104065    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m" event={"ID":"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f","Type":"ContainerStarted","Data":"1903ea0a592b581f5677db8c5e1a2850ad0c0a08d087b8faeb080b7801f1cccc"}
Oct 29 23:13:20.104167 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.104101    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m" event={"ID":"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f","Type":"ContainerStarted","Data":"9a6c5ebfb25c69ad015aa295eae4063df38e12f297e209d8b5709f81bef6e86f"}
Oct 29 23:13:20.105422 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.105390    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" event={"ID":"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3","Type":"ContainerStarted","Data":"fa00c09ca1049acdc8c6b9bed06d4395f5de534e394bdaeb1243763dd551bb8c"}
Oct 29 23:13:20.107658 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.107633    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkt7z" event={"ID":"9685c622-c264-4e53-8542-7a07650a6e84","Type":"ContainerStarted","Data":"ccd16946245dc8557053b2d6932e625fc0e0a1ba81e0c8992a5fdf77d6838ef8"}
Oct 29 23:13:20.110394 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.110368    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f24wt" event={"ID":"71eca11c-bf4c-473c-b64a-64041bbe808d","Type":"ContainerStarted","Data":"c26ca80185f874d61a07b5e01e7e2aa27a8b6b4cbb4b8fd23eb3b635d188ed3e"}
Oct 29 23:13:20.110470 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.110423    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f24wt" event={"ID":"71eca11c-bf4c-473c-b64a-64041bbe808d","Type":"ContainerStarted","Data":"0bd5aeb3d8e093ef30b6aa3bf24c95e481c9808879aa1b90b65bbbfa6614aeba"}
Oct 29 23:13:20.146155 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.146110    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mfwdq" podStartSLOduration=1.625343416 podStartE2EDuration="5.146097619s" totalImagesPullingTime="3.520754203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:20.145282222 +0000 UTC m=+48.949307534" watchObservedRunningTime="2026-10-29 23:13:20.146097619 +0000 UTC m=+48.950122919"
Oct 29 23:13:20.192065 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.192008    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gkt7z" podStartSLOduration=2.505960245 podStartE2EDuration="5.191990058s" totalImagesPullingTime="2.686029813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:13:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:20.190497622 +0000 UTC m=+48.994522925" watchObservedRunningTime="2026-10-29 23:13:20.191990058 +0000 UTC m=+48.996015360"
Oct 29 23:13:20.237697 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.237640    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m" podStartSLOduration=2.412955608 podStartE2EDuration="4.237621336s" totalImagesPullingTime="1.824665728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:20.237326006 +0000 UTC m=+49.041351307" watchObservedRunningTime="2026-10-29 23:13:20.237621336 +0000 UTC m=+49.041646638"
Oct 29 23:13:20.285613 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:20.285553    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f24wt" podStartSLOduration=2.976460838 podStartE2EDuration="4.285540072s" totalImagesPullingTime="1.21362424s" totalInitContainerRuntime="95.454994ms" isStatefulPod=true podCreationTimestamp="2026-10-29 23:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:20.284014608 +0000 UTC m=+49.088039921" watchObservedRunningTime="2026-10-29 23:13:20.285540072 +0000 UTC m=+49.089565373"
Oct 29 23:13:21.282659 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.282629    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"]
Oct 29 23:13:21.286672 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.286651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.307074 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.307052    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 29 23:13:21.307197 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.307181    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xvwxf\""
Oct 29 23:13:21.321352 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.321323    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"]
Oct 29 23:13:21.369794 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.369766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qnv\" (UniqueName: \"kubernetes.io/projected/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-kube-api-access-r8qnv\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.369958 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.369898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.470588 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.470499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.470588 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.470564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qnv\" (UniqueName: \"kubernetes.io/projected/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-kube-api-access-r8qnv\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.473039 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.473013    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.495376 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.495342    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qnv\" (UniqueName: \"kubernetes.io/projected/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-kube-api-access-r8qnv\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.597518 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.597482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:21.894173 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:21.894148    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"]
Oct 29 23:13:21.896224 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:21.896197    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd735fb1d_a5b9_4d8d_af87_bdaded8f82e6.slice/crio-3b6338c4814c92b312d18a5cc2ae34594eedbbc170877002d9bb1d8e92bb92bf WatchSource:0}: Error finding container 3b6338c4814c92b312d18a5cc2ae34594eedbbc170877002d9bb1d8e92bb92bf: Status 404 returned error can't find the container with id 3b6338c4814c92b312d18a5cc2ae34594eedbbc170877002d9bb1d8e92bb92bf
Oct 29 23:13:22.124094 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:22.124053    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" event={"ID":"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3","Type":"ContainerStarted","Data":"e7950391b9e16c6fb734447a77290c58ac4a51316efa688d35b717d0fa638b14"}
Oct 29 23:13:22.124240 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:22.124144    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" event={"ID":"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3","Type":"ContainerStarted","Data":"20aac285f3413d9019d076743546d52d03b0b0a892c48b72ff24010e76500fe0"}
Oct 29 23:13:22.124240 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:22.124191    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" event={"ID":"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3","Type":"ContainerStarted","Data":"0464a6c814e1b1b1f24468e7ccdd07031c73a51b4be682e30646b5b3167a9809"}
Oct 29 23:13:22.125229 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:22.125203    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs" event={"ID":"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6","Type":"ContainerStarted","Data":"3b6338c4814c92b312d18a5cc2ae34594eedbbc170877002d9bb1d8e92bb92bf"}
Oct 29 23:13:23.135550 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.135507    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" event={"ID":"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3","Type":"ContainerStarted","Data":"4bf276e9ac1f21a4fcea3d197ac51027947986203d65c575751c532bf0579424"}
Oct 29 23:13:23.135994 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.135624    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" event={"ID":"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3","Type":"ContainerStarted","Data":"d11d60779254b52a793eea53b062e394ba4cdfe0270a9dc8a353fddf1e9a0910"}
Oct 29 23:13:23.135994 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.135663    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" event={"ID":"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3","Type":"ContainerStarted","Data":"ff025ba51b2490efa4a29c16300e6230dd732ddd05fb8c0526d91c74ac8e5006"}
Oct 29 23:13:23.135994 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.135707    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:23.240901 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.240836    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg" podStartSLOduration=2.698799382 podStartE2EDuration="5.240821484s" totalImagesPullingTime="2.542022102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:23.227035252 +0000 UTC m=+52.031060544" watchObservedRunningTime="2026-10-29 23:13:23.240821484 +0000 UTC m=+52.044846784"
Oct 29 23:13:23.429851 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.429818    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:13:23.432558 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.432537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.452579 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.452557    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 23:13:23.452670 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.452589    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 23:13:23.452670 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.452590    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 23:13:23.466605 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466585    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 23:13:23.466743 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466726    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 23:13:23.466815 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466745    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 23:13:23.466815 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466754    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 23:13:23.466815 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466733    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 23:13:23.466815 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466750    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 23:13:23.466815 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466744    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 23:13:23.466815 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466796    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 23:13:23.466815 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466815    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 23:13:23.467287 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466822    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bk87z\""
Oct 29 23:13:23.467287 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466871    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 23:13:23.467287 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.466914    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 23:13:23.467287 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.467033    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6nojl5jppa58c\""
Oct 29 23:13:23.481158 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.481137    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:13:23.486698 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.486756 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.486756 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.486833 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.486833 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.486913 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.486943 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.486987 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487023 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.486996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487023 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487087 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487087 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487210 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487262 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487291 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487347 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487392 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487392 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487392 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.487515 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.487431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tms9l\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.557960 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.557929    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 23:13:23.588594 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.588740 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.588740 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.588740 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.588740 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.588995 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.588995 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.589095 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.588991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.589205 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.589305 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.589399 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.589565 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tms9l\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589448    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589798    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590043 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590737 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.590610    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590822 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.590748    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.590969 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.590940    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.591041 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.589553    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.591532 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.591506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.592092 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.592039    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.592255 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.592211    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.592949 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.592843    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.593181 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.593129    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.593904 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.593552    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.593904 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.593761    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.594180 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.594161    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.594180 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.594171    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.594276 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.594256    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.594665 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.594651    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.594701 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.594667    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.595273 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.595254    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.628629 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.628602    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tms9l\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:23.749142 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:23.749069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:24.086027 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:24.085948    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:13:24.101269 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:24.101247    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18375e9b_7089_4b3a_b691_bd0ba7a3441d.slice/crio-e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df WatchSource:0}: Error finding container e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df: Status 404 returned error can't find the container with id e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df
Oct 29 23:13:24.139571 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:24.139544    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs" event={"ID":"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6","Type":"ContainerStarted","Data":"c0241f497810abbded9890a6f07e3c3e1ab1a726a40eb00573eb378e9001a233"}
Oct 29 23:13:24.139952 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:24.139734    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:24.140969 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:24.140943    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerStarted","Data":"e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df"}
Oct 29 23:13:24.145334 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:24.145315    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:24.182747 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:24.182696    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs" podStartSLOduration=1.937650776 podStartE2EDuration="3.182683348s" totalImagesPullingTime="1.245032572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:24.181906937 +0000 UTC m=+52.985932241" watchObservedRunningTime="2026-10-29 23:13:24.182683348 +0000 UTC m=+52.986708676"
Oct 29 23:13:25.146284 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:25.146248    2582 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="42e6348341b119e421c41f88183fdeaf7597925a3800a0555ecd6d94caea888f" exitCode=0
Oct 29 23:13:25.146711 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:25.146336    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"42e6348341b119e421c41f88183fdeaf7597925a3800a0555ecd6d94caea888f"}
Oct 29 23:13:27.028764 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:27.028736    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:29.113855 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.113820    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:29.147821 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.147794    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:29.164646 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.164608    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerStarted","Data":"90ca07a0667961a7947f9e82f8e94415ac6781ea44391df19e4f5cb198fc287c"}
Oct 29 23:13:29.164822 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.164689    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerStarted","Data":"667f5df4ba4fb0fcb0a69b672ae4ab239f509d152b73431a04bd9e0055872e9e"}
Oct 29 23:13:29.164822 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.164736    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerStarted","Data":"6ec853fb7226f9f743c7a4a2f93ad6e3596d5b4c666b7af90826f3ad970b2255"}
Oct 29 23:13:29.164822 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.164775    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerStarted","Data":"29e7cf4aad4bbcc502aabc64824d72707be2a8b1061694ad11c963e7ee745c98"}
Oct 29 23:13:29.164822 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.164816    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerStarted","Data":"1d69a5f12faa5024bd37239518a665d184b6f4a398c7dbe95abda92cdcb53728"}
Oct 29 23:13:29.165058 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.164854    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerStarted","Data":"26a6a38a1174c1c01a23621ad6894bcaa7e95dfd6f621937d3395dcb43a09818"}
Oct 29 23:13:29.310448 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:29.310375    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.404459592 podStartE2EDuration="6.3103527s" totalImagesPullingTime="3.887429614s" totalInitContainerRuntime="18.463494ms" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:29.309803076 +0000 UTC m=+58.113828398" watchObservedRunningTime="2026-10-29 23:13:29.3103527 +0000 UTC m=+58.114378001"
Oct 29 23:13:31.276234 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:31.276192    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:31.294994 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:31.294965    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:32.179545 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:32.179512    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:32.194473 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:32.194448    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:36.518037 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.517993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:36.518037 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.518037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:36.536619 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.536594    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 23:13:36.540925 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.540900    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:36.550140 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.550119    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 23:13:36.560296 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.560269    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:36.618680 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.618642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:36.634579 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.634548    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 23:13:36.654600 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.654571    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 23:13:36.663037 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.663009    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:36.786828 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.786753    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m6t5c\""
Oct 29 23:13:36.791946 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.791923    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:36.792850 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.792832    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z28mk\""
Oct 29 23:13:36.798610 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.798596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:36.799218 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.799204    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wscf4\""
Oct 29 23:13:36.804598 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:36.804577    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:37.112031 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:37.112007    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b97hf"]
Oct 29 23:13:37.113166 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:37.113078    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mkj2j"]
Oct 29 23:13:37.114558 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:37.114534    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534d0681_f061_4879_9e5e_f734eebfc7a4.slice/crio-e02ba45412bcdc851b5aa056f1980d594108da27319e11c9abcfbb25f5a5cb75 WatchSource:0}: Error finding container e02ba45412bcdc851b5aa056f1980d594108da27319e11c9abcfbb25f5a5cb75: Status 404 returned error can't find the container with id e02ba45412bcdc851b5aa056f1980d594108da27319e11c9abcfbb25f5a5cb75
Oct 29 23:13:37.115338 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:37.115268    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9192aec_64e7_4d0f_8066_e236dc75b87d.slice/crio-42d8db269e2fd9fecb28132876aefa6ee8abe5f543e0e273d44c14100dc003cf WatchSource:0}: Error finding container 42d8db269e2fd9fecb28132876aefa6ee8abe5f543e0e273d44c14100dc003cf: Status 404 returned error can't find the container with id 42d8db269e2fd9fecb28132876aefa6ee8abe5f543e0e273d44c14100dc003cf
Oct 29 23:13:37.134089 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:37.134070    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bc5z8"]
Oct 29 23:13:37.136185 ip-10-0-139-224 kubenswrapper[2582]: W1029 23:13:37.136164    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b2a03_5d09_4c44_bec0_ec66bdea6781.slice/crio-3374c66fc7267c1e879ef793820c2f83a4f46258115ba4052c409d6f868bff20 WatchSource:0}: Error finding container 3374c66fc7267c1e879ef793820c2f83a4f46258115ba4052c409d6f868bff20: Status 404 returned error can't find the container with id 3374c66fc7267c1e879ef793820c2f83a4f46258115ba4052c409d6f868bff20
Oct 29 23:13:37.202527 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:37.202492    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b97hf" event={"ID":"534d0681-f061-4879-9e5e-f734eebfc7a4","Type":"ContainerStarted","Data":"e02ba45412bcdc851b5aa056f1980d594108da27319e11c9abcfbb25f5a5cb75"}
Oct 29 23:13:37.203601 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:37.203570    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bc5z8" event={"ID":"1d0b2a03-5d09-4c44-bec0-ec66bdea6781","Type":"ContainerStarted","Data":"3374c66fc7267c1e879ef793820c2f83a4f46258115ba4052c409d6f868bff20"}
Oct 29 23:13:37.204706 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:37.204681    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mkj2j" event={"ID":"d9192aec-64e7-4d0f-8066-e236dc75b87d","Type":"ContainerStarted","Data":"42d8db269e2fd9fecb28132876aefa6ee8abe5f543e0e273d44c14100dc003cf"}
Oct 29 23:13:39.216413 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:39.216374    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bc5z8" event={"ID":"1d0b2a03-5d09-4c44-bec0-ec66bdea6781","Type":"ContainerStarted","Data":"29c5d77d724f8c73ac14ef06769b9eea4de50704910f7663d6a0a876d8a368a9"}
Oct 29 23:13:39.216942 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:39.216450    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bc5z8" event={"ID":"1d0b2a03-5d09-4c44-bec0-ec66bdea6781","Type":"ContainerStarted","Data":"56a8cb8af2f78957d7a4ab4f5a7531f5b55599c1f5d9a820e0b8b6113fc4b49a"}
Oct 29 23:13:39.264938 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:39.264856    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bc5z8" podStartSLOduration=66.236702307 podStartE2EDuration="1m7.264837951s" totalImagesPullingTime="1.028135644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:39.264118427 +0000 UTC m=+68.068143786" watchObservedRunningTime="2026-10-29 23:13:39.264837951 +0000 UTC m=+68.068863253"
Oct 29 23:13:41.232254 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:41.232221    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b97hf" event={"ID":"534d0681-f061-4879-9e5e-f734eebfc7a4","Type":"ContainerStarted","Data":"0768dbe56481d245cdc0a716047fe98cb545c8325216552b81495a4c64121d36"}
Oct 29 23:13:41.232763 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:41.232411    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:41.277554 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:41.277489    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b97hf" podStartSLOduration=66.177796743 podStartE2EDuration="1m9.277467987s" totalImagesPullingTime="3.099671244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:41.274994898 +0000 UTC m=+70.079020205" watchObservedRunningTime="2026-10-29 23:13:41.277467987 +0000 UTC m=+70.081493289"
Oct 29 23:13:41.833569 ip-10-0-139-224 kubenswrapper[2582]: I1029 23:13:41.833433    2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 23:13:41.833577 ip-10-0-139-224 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 23:13:41.848236 ip-10-0-139-224 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 23:13:41.848405 ip-10-0-139-224 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 23:13:41.848584 ip-10-0-139-224 systemd[1]: kubelet.service: Consumed 4.190s CPU time.
Oct 29 23:13:41.863614 ip-10-0-139-224 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 23:13:41.929179 ip-10-0-139-224 kubenswrapper[6203]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 23:13:41.929179 ip-10-0-139-224 kubenswrapper[6203]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 23:13:41.929179 ip-10-0-139-224 kubenswrapper[6203]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 23:13:41.929179 ip-10-0-139-224 kubenswrapper[6203]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 23:13:41.931279 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931193    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.931279 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931278    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.931279 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931282    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931287    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931292    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931296    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931301    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931304    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931307    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931309    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931312    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931315    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931318    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931324    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931327    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931329    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931332    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931335    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931337    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931339    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931342    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.931389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931345    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931347    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931350    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931353    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931356    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931359    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931361    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931364    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931367    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931369    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931373    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931375    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931378    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931381    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931383    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931386    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931389    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931391    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931394    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931397    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.931912 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931400    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931403    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931406    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931408    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931411    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931414    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931416    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931419    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931422    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931424    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931427    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931430    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931433    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931435    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931438    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931441    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931444    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931446    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931449    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931452    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.932423 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931455    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931457    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931460    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931463    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931466    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931469    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931471    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931474    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931477    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931480    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931483    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931485    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931489    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931494    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931497    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931500    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931503    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931506    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931508    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.932932 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931512    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931514    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931517    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931520    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931523    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931525    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931528    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931530    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931533    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931535    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931538    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931540    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931543    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931546    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931548    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931552    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931555    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931557    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931560    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.933393 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931562    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931565    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931567    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931570    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931574    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931577    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.931579    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932464    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932472    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932482    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932486    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932489    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932492    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932497    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932501    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932505    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932509    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932512    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932514    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.933871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932517    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932520    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932529    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932532    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932535    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932538    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932541    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932543    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932546    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932549    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932551    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932554    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932557    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932560    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932562    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932565    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932567    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932570    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932573    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932575    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.934364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932578    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932580    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932583    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932591    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932594    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932597    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932599    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932602    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932605    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932607    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932610    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932612    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932615    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932618    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932620    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932623    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932625    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932628    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932631    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.934833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932633    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932636    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932639    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932642    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932644    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932647    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932650    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932652    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932655    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932657    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932660    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932662    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932665    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932667    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932670    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932672    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932675    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932678    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932681    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.935341 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932683    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932686    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932688    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932691    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932693    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932696    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932698    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932701    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932703    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932706    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932709    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932711    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932714    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932717    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932720    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932723    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932725    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932728    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932730    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932733    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.935853 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932735    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932740    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932744    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932748    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932751    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932753    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932756    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932758    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932760    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932763    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932765    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932768    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932771    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932773    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932776    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.932779    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933289    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933295    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933300    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933303    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.936389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933306    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933309    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933312    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933314    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933317    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933319    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933325    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933328    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933331    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933333    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933336    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933339    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933341    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933344    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933346    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933349    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933352    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933354    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933357    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933359    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.936856 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933362    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933366    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933370    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933373    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933377    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933379    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933382    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933384    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933387    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933390    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933392    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933395    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933398    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933401    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933404    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933406    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933409    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933412    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933416    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.937403 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933419    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933422    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933425    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933427    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933430    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933432    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933435    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933438    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933441    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933444    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933446    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933449    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933452    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933454    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933457    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933460    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933462    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933465    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933468    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933470    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.937921 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933476    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933479    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933481    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933485    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933487    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933490    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933492    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933495    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933498    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933500    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933503    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933508    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933511    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933513    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933516    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933518    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933521    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933524    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933526    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.938421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933529    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933531    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933534    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933536    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933539    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933541    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933544    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933546    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933549    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933551    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933555    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933559    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933563    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933568    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933570    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933573    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933575    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933578    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933581    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.938887 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933583    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933587    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933590    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933592    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.933595    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933696    6203 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933707    6203 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933716    6203 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933723    6203 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933728    6203 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933732    6203 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933736    6203 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933740    6203 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933744    6203 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933747    6203 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933751    6203 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933754    6203 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933757    6203 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933760    6203 flags.go:64] FLAG: --cgroup-root=""
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933763    6203 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933767    6203 flags.go:64] FLAG: --client-ca-file=""
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933770    6203 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933773    6203 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933778    6203 flags.go:64] FLAG: --cluster-domain=""
Oct 29 23:13:41.939407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933781    6203 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933785    6203 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933790    6203 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933794    6203 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933798    6203 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933801    6203 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933804    6203 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933808    6203 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933811    6203 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933814    6203 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933817    6203 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933820    6203 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933823    6203 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933830    6203 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933833    6203 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933837    6203 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933840    6203 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933843    6203 flags.go:64] FLAG: --enable-server="true"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933846    6203 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933851    6203 flags.go:64] FLAG: --event-burst="100"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933854    6203 flags.go:64] FLAG: --event-qps="50"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933857    6203 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933860    6203 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933863    6203 flags.go:64] FLAG: --eviction-hard=""
Oct 29 23:13:41.940353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933868    6203 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933871    6203 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933885    6203 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933889    6203 flags.go:64] FLAG: --eviction-soft=""
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933892    6203 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933896    6203 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933899    6203 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933902    6203 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933906    6203 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933908    6203 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933914    6203 flags.go:64] FLAG: --feature-gates=""
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933918    6203 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933922    6203 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933925    6203 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933928    6203 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933931    6203 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933935    6203 flags.go:64] FLAG: --help="false"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933938    6203 flags.go:64] FLAG: --hostname-override="ip-10-0-139-224.ec2.internal"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933941    6203 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933944    6203 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933948    6203 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933953    6203 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933956    6203 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933959    6203 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 23:13:41.941034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933962    6203 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933965    6203 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933968    6203 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933971    6203 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933974    6203 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933977    6203 flags.go:64] FLAG: --kube-reserved=""
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933980    6203 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933983    6203 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933987    6203 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933990    6203 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933992    6203 flags.go:64] FLAG: --lock-file=""
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933996    6203 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.933999    6203 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934002    6203 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934008    6203 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934011    6203 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934015    6203 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934018    6203 flags.go:64] FLAG: --logging-format="text"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934022    6203 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934027    6203 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934030    6203 flags.go:64] FLAG: --manifest-url=""
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934033    6203 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934037    6203 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934040    6203 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934044    6203 flags.go:64] FLAG: --max-pods="110"
Oct 29 23:13:41.941625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934048    6203 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934051    6203 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934054    6203 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934057    6203 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934061    6203 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934064    6203 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934067    6203 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934075    6203 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934078    6203 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934081    6203 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934084    6203 flags.go:64] FLAG: --pod-cidr=""
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934087    6203 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934090    6203 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934093    6203 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934096    6203 flags.go:64] FLAG: --port="10250"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934099    6203 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934103    6203 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c129f8b72838dc6a"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934106    6203 flags.go:64] FLAG: --qos-reserved=""
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934109    6203 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934112    6203 flags.go:64] FLAG: --register-node="true"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934115    6203 flags.go:64] FLAG: --register-with-taints=""
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934119    6203 flags.go:64] FLAG: --registry-burst="10"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934122    6203 flags.go:64] FLAG: --registry-qps="5"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934125    6203 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934128    6203 flags.go:64] FLAG: --reserved-memory=""
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934133    6203 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 23:13:41.942243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934137    6203 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934141    6203 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934144    6203 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934147    6203 flags.go:64] FLAG: --runonce="false"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934150    6203 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934153    6203 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934156    6203 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934159    6203 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934162    6203 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934165    6203 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934170    6203 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934173    6203 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934176    6203 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934179    6203 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934182    6203 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934185    6203 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934188    6203 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934191    6203 flags.go:64] FLAG: --system-cgroups=""
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934194    6203 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934200    6203 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934202    6203 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934205    6203 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934210    6203 flags.go:64] FLAG: --tls-min-version=""
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934213    6203 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934216    6203 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 23:13:41.942883 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934219    6203 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934221    6203 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934224    6203 flags.go:64] FLAG: --v="2"
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934229    6203 flags.go:64] FLAG: --version="false"
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934233    6203 flags.go:64] FLAG: --vmodule=""
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934237    6203 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934242    6203 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934387    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934391    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934398    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934401    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934404    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934406    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934409    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934412    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934414    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934417    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934420    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934423    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934425    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934428    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.943462 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934431    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934433    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934436    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934439    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934441    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934444    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934447    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934450    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934452    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934455    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934458    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934460    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934463    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934466    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934468    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934471    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934473    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934478    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934481    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934484    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.943973 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934486    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934489    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934492    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934494    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934497    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934500    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934502    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934505    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934507    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934510    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934513    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934515    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934518    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934521    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934523    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934526    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934528    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934531    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934534    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.944517 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934538    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934540    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934543    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934545    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934548    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934550    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934553    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934555    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934558    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934560    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934565    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934568    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934570    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934573    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934576    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934579    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934582    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934585    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934587    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934590    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.945296 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934593    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934595    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934598    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934601    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934603    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934606    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934609    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934611    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934614    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934618    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934622    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934625    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934628    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934630    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934633    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934635    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934638    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934640    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934643    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.945833 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934646    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934649    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934652    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934656    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934658    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934661    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934664    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934666    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934669    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934672    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934675    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934677    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934680    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934682    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.934710    6203 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934746    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.946386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934750    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934753    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934756    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934759    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934762    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934765    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934767    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934770    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934773    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934775    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934778    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934781    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934783    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934786    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934788    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934791    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934793    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934796    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934798    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934801    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.946839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934806    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934808    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934811    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934814    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934816    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934821    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934824    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934826    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934829    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934832    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934834    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934838    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934843    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934846    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934849    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934852    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934854    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934857    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934859    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.947394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934862    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934865    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934867    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934870    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934888    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934891    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934894    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934896    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934899    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934902    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934904    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934907    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934910    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934913    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934916    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934918    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934921    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934923    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934928    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.947926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934930    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934933    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934936    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934940    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934944    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934947    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934950    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934952    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934955    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934958    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934961    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934963    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934966    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934969    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934971    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934974    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934977    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934979    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934982    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.948420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934985    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934988    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934991    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934993    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934996    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.934998    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935001    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935003    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935006    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935009    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935011    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935014    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935018    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935020    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935023    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935026    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935028    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935031    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935033    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935036    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.948904 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935038    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935041    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935043    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935046    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935048    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935051    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935053    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.935056    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.938842    6203 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.938858    6203 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.938896    6203 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.938904    6203 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938963    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938969    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938972    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938976    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938978    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938981    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938984    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938987    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.949404 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938991    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938996    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.938999    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939002    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939005    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939007    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939010    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939013    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939015    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939018    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939021    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939023    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939026    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939029    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939033    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939037    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939040    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939043    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939045    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.949897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939049    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939051    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939056    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939059    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939062    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939065    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939068    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939070    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939073    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939075    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939078    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939081    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939084    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939086    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939089    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939092    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939094    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939097    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939099    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939102    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.950364 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939105    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939107    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939110    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939112    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939115    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939118    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939120    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939123    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939125    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939128    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939130    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939133    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939136    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939139    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939141    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939145    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939147    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939150    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939152    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.950897 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939155    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939157    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939160    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939162    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939165    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939168    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939170    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939173    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939175    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939178    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939181    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939183    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939186    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939188    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939191    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939193    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939196    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939198    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939201    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939204    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.951363 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939206    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939209    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939212    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939214    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939217    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939219    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939222    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939224    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939227    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939230    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939232    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939235    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939238    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939240    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939243    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939246    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939248    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939251    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939254    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.951839 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939256    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.939286    6203 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939948    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939965    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939971    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939976    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939981    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939986    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939991    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939995    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.939999    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940004    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940014    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940020    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940024    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940029    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.952325 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940033    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940038    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940042    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940047    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940057    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940063    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940067    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940073    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940085    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940095    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940099    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940104    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940109    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940113    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940118    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940122    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940126    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940131    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940135    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940139    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.952714 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940143    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940148    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940159    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940165    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940170    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940175    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940179    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940183    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940188    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940192    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940196    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940202    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940206    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940210    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940219    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940224    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940229    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940234    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940238    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.953213 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940243    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940248    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940251    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940254    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940257    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940260    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940263    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940265    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940273    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940277    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940281    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940284    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940288    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940293    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940297    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940301    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940305    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940310    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940314    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.953670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940319    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940327    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940332    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940336    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940340    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940344    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940349    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940353    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940358    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940363    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940368    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940373    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940377    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940382    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940391    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940396    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940400    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940405    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940409    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.954176 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940413    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940417    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940421    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940425    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940430    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940434    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940439    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940448    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940452    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940456    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940461    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940465    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940469    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940473    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940478    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.954643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.940537    6203 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940755    6203 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940764    6203 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940768    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940771    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940775    6203 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940779    6203 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940781    6203 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940784    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940787    6203 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940790    6203 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940793    6203 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940795    6203 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940798    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940802    6203 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940805    6203 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940808    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940811    6203 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940813    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 23:13:41.955021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940816    6203 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940819    6203 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940822    6203 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940825    6203 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940828    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940831    6203 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940834    6203 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940836    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940839    6203 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940842    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940845    6203 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940848    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940851    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940854    6203 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940857    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940860    6203 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940862    6203 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940865    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940868    6203 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940870    6203 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 23:13:41.955480 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940873    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940893    6203 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940897    6203 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940902    6203 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940905    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940910    6203 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940913    6203 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940916    6203 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940919    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940922    6203 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940925    6203 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940928    6203 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940931    6203 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940934    6203 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940936    6203 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940939    6203 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940942    6203 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940945    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940947    6203 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940950    6203 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 23:13:41.956015 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940953    6203 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940955    6203 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940958    6203 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940961    6203 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940964    6203 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940967    6203 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940970    6203 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940972    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940977    6203 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940981    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940984    6203 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940987    6203 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940990    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940993    6203 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940996    6203 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.940998    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941001    6203 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941004    6203 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941007    6203 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 23:13:41.957802 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941014    6203 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941017    6203 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941020    6203 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941023    6203 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941025    6203 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941028    6203 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941031    6203 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941034    6203 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941036    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941039    6203 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941042    6203 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941045    6203 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941048    6203 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941051    6203 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941055    6203 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941059    6203 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941062    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941065    6203 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941068    6203 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 23:13:41.958411 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941072    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941075    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941078    6203 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941080    6203 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941083    6203 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941085    6203 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941088    6203 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941091    6203 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941093    6203 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:13:41.941096    6203 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.941165    6203 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.942141    6203 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.942191    6203 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.942374    6203 server.go:1014] "Starting client certificate rotation"
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.942515    6203 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.942566    6203 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 23:07:32 +0000 UTC" deadline="2028-08-08 21:57:58.504697586 +0000 UTC"
Oct 29 23:13:41.958872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.942596    6203 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15574h44m16.562103845s"
Oct 29 23:13:41.959299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.942828    6203 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 23:13:41.959299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.943931    6203 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 23:13:41.959299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.945202    6203 log.go:25] "Validated CRI v1 runtime API"
Oct 29 23:13:41.959299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.947092    6203 log.go:25] "Validated CRI v1 image API"
Oct 29 23:13:41.959299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.947806    6203 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 23:13:41.959299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.949915    6203 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 807f0546-93dd-4376-81f1-2270604a75c1:/dev/nvme0n1p3 f74100c9-14a0-46ff-aea3-8ff38ce07abd:/dev/nvme0n1p4]
Oct 29 23:13:41.959477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.949937    6203 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03c425d6a4884fb9ef45474864d654a9e76987884b5d7bfa8928d7b05b1829a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03c425d6a4884fb9ef45474864d654a9e76987884b5d7bfa8928d7b05b1829a6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3374c66fc7267c1e879ef793820c2f83a4f46258115ba4052c409d6f868bff20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3374c66fc7267c1e879ef793820c2f83a4f46258115ba4052c409d6f868bff20/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b6338c4814c92b312d18a5cc2ae34594eedbbc170877002d9bb1d8e92bb92bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b6338c4814c92b312d18a5cc2ae34594eedbbc170877002d9bb1d8e92bb92bf/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42d8db269e2fd9fecb28132876aefa6ee8abe5f543e0e273d44c14100dc003cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42d8db269e2fd9fecb28132876aefa6ee8abe5f543e0e273d44c14100dc003cf/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dc065a6af4c6476a6fe945a192375793bec2252fe5613ed0273994d4d0a775a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dc065a6af4c6476a6fe945a192375793bec2252fe5613ed0273994d4d0a775a/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f205fc76786280c8e65bfa2041c42913ac553f359e25fd0039b651f5e6a292c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f205fc76786280c8e65bfa2041c42913ac553f359e25fd0039b651f5e6a292c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/863826145f66d91883bb1609c0c8a89a873b6f661d014dd8dc326d87cea21c3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/863826145f66d91883bb1609c0c8a89a873b6f661d014dd8dc326d87cea21c3e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d1ffca31f6f8bdb3bc7c0fc1f1e02ceda76c8883a16ea07fe4ccbf0c5eff5ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d1ffca31f6f8bdb3bc7c0fc1f1e02ceda76c8883a16ea07fe4ccbf0c5eff5ec/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a259acccbe5603c4abee3e6957f5a1c5af92e311b33c144595f79c10128ae777/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a259acccbe5603c4abee3e6957f5a1c5af92e311b33c144595f79c10128ae777/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad9e327416fb0dbce28d069d845d6aaede4b72ffc5b931448ab44b95af4fffd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad9e327416fb0dbce28d069d845d6aaede4b72ffc5b931448ab44b95af4fffd3/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf86a9fc8d17711a7fdd5ad341258cef7fa9bfc867dbfc5cfcf881e419300acc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf86a9fc8d17711a7fdd5ad341258cef7fa9bfc867dbfc5cfcf881e419300acc/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e02ba45412bcdc851b5aa056f1980d594108da27319e11c9abcfbb25f5a5cb75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e02ba45412bcdc851b5aa056f1980d594108da27319e11c9abcfbb25f5a5cb75/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0728485e480488df5fc6c28d939f74a0e94198481fede9f0e4d51038954d6ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0728485e480488df5fc6c28d939f74a0e94198481fede9f0e4d51038954d6ce/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e17457179617ce890e6b15d1779d84da3d51cf3f190d3b53a92b535ea17f9317/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e17457179617ce890e6b15d1779d84da3d51cf3f190d3b53a92b535ea17f9317/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3757c3a68734d6ffc6ca1cf6f366e5a6f2b1bfb85d352789488d19bfb18ee2a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3757c3a68734d6ffc6ca1cf6f366e5a6f2b1bfb85d352789488d19bfb18ee2a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e92b562f3610f65280cd7226e8e09a8ad1906335a4ba02f4d2313792519be2af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e92b562f3610f65280cd7226e8e09a8ad1906335a4ba02f4d2313792519be2af/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eac0e60ef6423aae5e7665e0b612dacf8bda3814d966d1b3d030c7852e9fd241/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eac0e60ef6423aae5e7665e0b612dacf8bda3814d966d1b3d030c7852e9fd241/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edd42781e655509d8bf4b48fee36140e35717a2b3d3531bca0514643d7a95c1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edd42781e655509d8bf4b48fee36140e35717a2b3d3531bca0514643d7a95c1c/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa00c09ca1049acdc8c6b9bed06d4395f5de534e394bdaeb1243763dd551bb8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa00c09ca1049acdc8c6b9bed06d4395f5de534e394bdaeb1243763dd551bb8c/userdata/shm major:0 minor:280 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1769c496-3b59-4aba-a674-13dae505ed09/volumes/kubernetes.io~projected/kube-api-access-5k7k8:{mountpoint:/var/lib/kubelet/pods/1769c496-3b59-4aba-a674-13dae505ed09/volumes/kubernetes.io~projected/kube-api-access-5k7k8 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1769c496-3b59-4aba-a674-13dae505ed09/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1769c496-3b59-4aba-a674-13dae505ed09/volumes/kubernetes.io~secret/metrics-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/config-reloader/2 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/prometheus/15 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~projected/kube-api-access-tms9l:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~projected/kube-api-access-tms9l major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~projected/tls-assets major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/config major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/web-config major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d0b2a03-5d09-4c44-bec0-ec66bdea6781/volumes/kubernetes.io~projected/kube-api-access-8fd66:{mountpoint:/var/lib/kubelet/pods/1d0b2a03-5d09-4c44-bec0-ec66bdea6781/volumes/kubernetes.io~projected/kube-api-access-8fd66 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d0b2a03-5d09-4c44-bec0-ec66bdea6781/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1d0b2a03-5d09-4c44-bec0-ec66bdea6781/volumes/kubernetes.io~secret/metrics-certs major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volumes/kubernetes.io~projected/kube-api-access-prn7r:{mountpoint:/var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volumes/kubernetes.io~projected/kube-api-access-prn7r major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a24e0c9-e7a1-4206-bdb5-a956b226a79e/volumes/kubernetes.io~projected/kube-api-access-94g2q:{mountpoint:/var/lib/kubelet/pods/2a24e0c9-e7a1-4206-bdb5-a956b226a79e/volumes/kubernetes.io~projected/kube-api-access-94g2q major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e327984-d0fc-4330-a933-f104718bce7c/volumes/kubernetes.io~projected/kube-api-access-tqr78:{mountpoint:/var/lib/kubelet/pods/2e327984-d0fc-4330-a933-f104718bce7c/volumes/kubernetes.io~projected/kube-api-access-tqr78 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/534d0681-f061-4879-9e5e-f734eebfc7a4/volumes/kubernetes.io~projected/kube-api-access-26l5f:{mountpoint:/var/lib/kubelet/pods/534d0681-f061-4879-9e5e-f734eebfc7a4/volumes/kubernetes.io~projected/kube-api-access-26l5f major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~projected/kube-api-access-fbshl:{mountpoint:/var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~projected/kube-api-access-fbshl major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~projected/kube-api-access-vw22s:{mountpoint:/var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~projected/kube-api-access-vw22s major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~projected/kube-api-access-zm8cb:{mountpoint:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~projected/kube-api-access-zm8cb major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89e02182-3b19-41c8-9040-f75a4254187b/volumes/kubernetes.io~projected/kube-api-access-kgcz9:{mountpoint:/var/lib/kubelet/pods/89e02182-3b19-41c8-9040-f75a4254187b/volumes/kubernetes.io~projected/kube-api-access-kgcz9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9685c622-c264-4e53-8542-7a07650a6e84/volumes/kubernetes.io~projected/kube-api-access-vgrh6:{mountpoint:/var/lib/kubelet/pods/9685c622-c264-4e53-8542-7a07650a6e84/volumes/kubernetes.io~projected/kube-api-access-vgrh6 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9685c622-c264-4e53-8542-7a07650a6e84/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/9685c622-c264-4e53-8542-7a07650a6e84/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96cf9d61-5fb4-45ed-b68b-868a6738f12c/volumes/kubernetes.io~projected/kube-api-access-nzqjq:{mountpoint:/var/lib/kubelet/pods/96cf9d61-5fb4-45ed-b68b-868a6738f12c/volumes/kubernetes.io~projected/kube-api-access-nzqjq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~projected/kube-api-access-r88bf:{mountpoint:/var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~projected/kube-api-access-r88bf major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~projected/kube-api-access-rgbtb:{mountpoint:/var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~projected/kube-api-access-rgbtb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6/volumes/kubernetes.io~projected/kube-api-access-r8qnv:{mountpoint:/var/lib/kubelet/pods/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6/volumes/kubernetes.io~projected/kube-api-access-r8qnv major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9192aec-64e7-4d0f-8066-e236dc75b87d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d9192aec-64e7-4d0f-8066-e236dc75b87d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9a2e31a-049d-48d2-979c-8557f7c239f2/volumes/kubernetes.io~projected/kube-api-access-dfxp2:{mountpoint:/var/lib/kubelet/pods/d9a2e31a-049d-48d2-979c-8557f7c239f2/volumes/kubernetes.io~projected/kube-api-access-dfxp2 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f643dab0-17a4-410b-b56b-f278862fa237/volumes/kubernetes.io~projected/kube-api-access-lpflj:{mountpoint:/var/lib/kubelet/pods/f643dab0-17a4-410b-b56b-f278862fa237/volumes/kubernetes.io~projected/kube-api-access-lpflj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f88a9388-d3c1-421a-bb58-87a481f7333a/volumes/kubernetes.io~projected/kube-api-access-mtk22:{mountpoint:/var/lib/kubelet/pods/f88a9388-d3c1-421a-bb58-87a481f7333a/volumes/kubernetes.io~projected/kube-api-access-mtk22 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f88a9388-d3c1-421a-bb58-87a481f7333a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f88a9388-d3c1-421a-bb58-87a481f7333a/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/faf7f273-a82a-40a5-811c-efb7281d1db6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/faf7f273-a82a-40a5-811c-efb7281d1db6/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/368ccedfa826e1da9da718a4655cd567703d9ce3cc49a335baeff1f7213dd927/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bdd7dabc1e9d43fc93fe9f8ef0091cd7d4fc2c7278d3a5b2c7801dba555b4eb7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f92b69c211dd11a2a608ee498d6999ecb5a7d0aa89841b6b9491218c7b9a85c0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/fb011b119b5abb5372795010229ce66b543002edfa995651ee253482aca59ee5/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b435ddb9dc585a08ae63c205d4785a080c28ce5dc8cfc9fadfe1a2be5aa95a5d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ae8671b751cc181c13e80fa6651eaf69ffb47a36b1f34b3281c5022720e93d0e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fc6c22714894270a09370d38f9bc79cd95b09e7dc18693b8615170f7e0cd40d6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/dde17aeb09219381500616302c6f01af721fdfba038d91b77eca24c747afcd85/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b0ea3b91c15b8aa10434c44d5cc4a54c71807c4d395f3d8818668451b12f48f3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ba9cb9624725886a46da3f355eabe01a3e2156e2891190a0ba96e889306c9be2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/52aa3e5e39e36d53a46ebbdc6c30859c3df0ce282d1cc4876c69d2c73af9dfe2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c9f30d50dee8bf723f87d8a09bb776ca8dca8e7719f06a9b6c55b6e4422dc605/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-150:{mountpoint:/var/lib/containers/storage/overlay/8c19521da3283ae9c6965f3fc5eb8d1dde4a25f2f3d2e9f675a794d7a5cc9633/merged major:0 minor:150 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/91d1f9ac91338b78af4e322efbc476b3b02b5dd6f351c33cc1ceae492d583306/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/deb9cd03a586d991a50594705dfa62605a94fff2185aef8dd2584d0b314e0c21/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/71f54264099f5eeea2da90c0b11fea41a6f580cb8b77e4764e381697edd37c8d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3fca036531687b27d52dff5ff2b14a6cfafe8566b491332b2381b241a8b4e5b2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/beccabea2b34e92e32dd0f1414d3bceebfe2f7950bc669c90dd3dd9f689d00d0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5f51fee368d1798de678947c9564019695c0f15ae22cc359d9193b401666f905/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2ea945677f9156185c0bbd1fb3a4dcddfe156a59a9d583338490d067c3aeb369/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2d6af96126319bcc6605dbc3cae91441000253e343b70d9a0298709a9fd8593b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8a1fcd2d8a84272101b46f07e7eb953cd06e0eb1520752718f84ccaf1b8e0f82/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/6e267b47e0bd466990b53c2027ba7d236cf786ef0519b3cdfe0ce541dbf09900/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/eee472aa35c441bd2a3745dcc792765b98c81af01a7c8f37562bb10479b9c3da/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/2f16b5848bdcc603253b4657a282a5ff3c0a8114575a1bc7e97e73d45c14f351/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/56e12a27e412d1658744f56911ee7cface262094586b5b038042cda7547c39d1/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/0096fad14191a0da85dc8326e1c882c3e70272fb12ca09d2fb35fc086e44975e/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/5435cef60b71c4f701be979d6a30dde907d8d9f364476723a171652bb701ced6/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/73b88087d51712d8dca46c50cf9f0f80055a0adefe5a47ae77b104e5fc0eedb6/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/28278fa31c82adfc6e69377585faf7b3c16488a899bf50433b8c62a05dfc58ab/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/69204fe1049e3e8e65c78b78b1c5d17ec20082b50b2f61499effd6fb3d3eba41/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/63d8a0cffb4791ae96af8e3564fa877f2c095cba981415d897b0474170c3b7de/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/6eafba62d9112b9f2ea562fd365c2ab7452c8508c1857a37fd7c598534a249a5/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/3d59b39e77db7533af94299ccc5d761ec868f0eabedf7d386df71d252fbd90f9/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/965f7c1fb9534d7ce5b2e0a1b9c8239c0f21050ab4a2c8d798aac49db32f2f63/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/baf09178ebe75b4ef9d01f07b84f4d7f476a208a571b0319efeb25b3d21d96f8/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/9c8fd0a02dd7780b97d1960845574e6c3915e47d76bd8702018a6297f6bbdb4d/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/9a24ab799b45e19dc917a8c4e25d3a0ed06a844a89f0b37f3b8e916519a66127/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/aa166a0431e9f21741ac624b6b47485dca035da2e1215a67602a3744aa266105/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/bee78bc51f3f2a21261cee9e46b0b523ebe62ca885f18c9d94c8f8d5f7be8726/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/9cfc18c14bb1d5afa9c90aa7b3d2373864782fc1896968a521b0a778ba5487c5/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/e9e66f04cff9757490cf399ad63501f984271aa42bd63d61783a5e773194acdb/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/6015029c9afcfa02c4a9368d9a0cafa85d510d5e201495d14a80d3aead86a520/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/8abbbfb9f67e9e99c0b6a3d60c13b70a2d5f91a053d7912c914661e0bdb4fd25/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/6a142cf65bacd267e7b957af7e2755368b6371fcdb784a2285761c7d471a80f3/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/2107e33b60012681045ba82ce65065e5c92434f3d64571ab708a040d48158108/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/e3fc06b95a2e47297da2894b842ccb1e629d06af055232ee54040f7130feac74/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/d817637987b228b2a7370c61791ef0baf3172ca55dfbb36dcd634bf2c304390c/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/738b883a2d4796aca1db3275e11dbfd302c2919b3fd9465d3dcff79874812d25/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/24bc2fdb02992ada1cbabd63a88101945d2ea15aac6360f4db8686b8c4872c19/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/64869cd1d185d93c0f14899b8cc03e53122d236cde800b261091b44b8a865b7b/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/4847acfc91b23e6c3e36888f26e78d1c20c30fcb74f7a9b4cd852f5790893633/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/9b749ccd6a5fea999bf499bbf0291077dcef0a1620db8a7f100937eff6904228/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/a1bd93e803a61d5161402f17447c5ecfb1e4740e0712d6e192f91dfaca4c72e5/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/df692a00aaaed103ffc5bcb80672edc5e65486a54f6688e7fee5dcdf7a086c67/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/e3e0a10d02e5532a92bdc4aabed1923cbef0e8c1d78ebc773079ff79a026e8d2/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/30dadc442ee5f18e077193e7f3736e2077117f848b5330d2a3d7df77d72f2b7d/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/23019ea8a6ae16e5da5cfd762c3b15f9b3bd909dee22d78bc54d9129fea943e9/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/2e092c320850846d02dc99ea6b11eb7974e6db135ddb58986e4a47d32d785e34/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/b12fc194154d44a80f3da632907a8c0caad6c57df9096916788305b0cd95b80a/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/270b27889449455cff34b8e6ffa7956dbf0af42892f105608a0433def5db2fd0/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/27afadf7f4173cc9961abb9cf307b974b60d6344fad9323c068f5f2ee4d807f7/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/1afb83fedaee99427d7bad24694f01a59f22889b674769375f7555b0aef79531/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/9eed0a81083199548ed579d0c3cbc94e508aeedb9f0703b2522985b4d7c4a83f/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0c2d4c8e9d0c4e31ace3f90d72ad7d816b26a44730885432058778999a31a503/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9ca40f13b0765f2e5c8a52910b535274c9bd4e37934e50074bce994dd03995f0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b406438eb77981bc6402a7243a89e56ef16e0dc3a26e8a0093753efcbf91ade4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/34408dd193df9c83766bf4820b8ed0ca4453f6bb7c02c1bd6ade703d465ecf0a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/26ef2ffc190dd3aabfba4605ff886ef1452bb42410393bf4ec4e79c3f3d809b7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c91e76ef83bd8380ed6d90247d824d00899bff78de3f6d282fe11dfa72a3162c/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6b3e965c14bf902c082f18091b51f0f115c875445ff7a9bbe8eed35625e43ac0/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/004929a3312e1edb58231bd81f22fba23a7620330148afda02a1e5f1a214b116/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5405530ac368711814c494f8854e74ab72a26cd975e56f2ac8fef41bbfdde693/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/31ed6be551a37b206adf3fca9890143b4678bcd4c8c608a1d2abd72892234f59/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.958649    6203 manager.go:217] Machine: {Timestamp:2026-10-29 23:13:41.958125306 +0000 UTC m=+0.062385683 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100148 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21633e0ed6a78a235cdd62784fe1e5 SystemUUID:ec21633e-0ed6-a78a-235c-dd62784fe1e5 BootID:fe0e5392-2ca4-4e8e-a139-24b98ae44d9f Filesystems:[{Device:/var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~projected/kube-api-access-r88bf DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3374c66fc7267c1e879ef793820c2f83a4f46258115ba4052c409d6f868bff20/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9685c622-c264-4e53-8542-7a07650a6e84/volumes/kubernetes.io~projected/kube-api-access-vgrh6 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~projected/kube-api-access-zm8cb DeviceMajor:0 DeviceMinor:279 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/faf7f273-a82a-40a5-811c-efb7281d1db6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f205fc76786280c8e65bfa2041c42913ac553f359e25fd0039b651f5e6a292c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d0b2a03-5d09-4c44-bec0-ec66bdea6781/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:411 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89e02182-3b19-41c8-9040-f75a4254187b/volumes/kubernetes.io~projected/kube-api-access-kgcz9 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~projected/kube-api-access-rgbtb DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9685c622-c264-4e53-8542-7a07650a6e84/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42d8db269e2fd9fecb28132876aefa6ee8abe5f543e0e273d44c14100dc003cf/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-150 DeviceMajor:0 DeviceMinor:150 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~projected/kube-api-access-vw22s DeviceMajor:0 DeviceMinor:231 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b6338c4814c92b312d18a5cc2ae34594eedbbc170877002d9bb1d8e92bb92bf/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a259acccbe5603c4abee3e6957f5a1c5af92e311b33c144595f79c10128ae777/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf86a9fc8d17711a7fdd5ad341258cef7fa9bfc867dbfc5cfcf881e419300acc/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e02ba45412bcdc851b5aa056f1980d594108da27319e11c9abcfbb25f5a5cb75/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03c425d6a4884fb9ef45474864d654a9e76987884b5d7bfa8928d7b05b1829a6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6/volumes/kubernetes.io~projected/kube-api-access-r8qnv DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:368 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:376 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f643dab0-17a4-410b-b56b-f278862fa237/volumes/kubernetes.io~projected/kube-api-access-lpflj DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/534d0681-f061-4879-9e5e-f734eebfc7a4/volumes/kubernetes.io~projected/kube-api-access-26l5f DeviceMajor:0 DeviceMinor:416 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e327984-d0fc-4330-a933-f104718bce7c/volumes/kubernetes.io~projected/kube-api-access-tqr78 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d1ffca31f6f8bdb3bc7c0fc1f1e02ceda76c8883a16ea07fe4ccbf0c5eff5ec/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa00c09ca1049acdc8c6b9bed06d4395f5de534e394bdaeb1243763dd551bb8c/userdata/shm DeviceMajor:0 DeviceMinor:280 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f88a9388-d3c1-421a-bb58-87a481f7333a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e17457179617ce890e6b15d1779d84da3d51cf3f190d3b53a92b535ea17f9317/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1769c496-3b59-4aba-a674-13dae505ed09/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad9e327416fb0dbce28d069d845d6aaede4b72ffc5b931448ab44b95af4fffd3/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eac0e60ef6423aae5e7665e0b612dacf8bda3814d966d1b3d030c7852e9fd241/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3757c3a68734d6ffc6ca1cf6f366e5a6f2b1bfb85d352789488d19bfb18ee2a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0728485e480488df5fc6c28d939f74a0e94198481fede9f0e4d51038954d6ce/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edd42781e655509d8bf4b48fee36140e35717a2b3d3531bca0514643d7a95c1c/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1769c496-3b59-4aba-a674-13dae505ed09/volumes/kubernetes.io~projected/kube-api-access-5k7k8 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dc065a6af4c6476a6fe945a192375793bec2252fe5613ed0273994d4d0a775a/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~projected/kube-api-access-tms9l DeviceMajor:0 DeviceMinor:378 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9a2e31a-049d-48d2-979c-8557f7c239f2/volumes/kubernetes.io~projected/kube-api-access-dfxp2 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e92b562f3610f65280cd7226e8e09a8ad1906335a4ba02f4d2313792519be2af/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/volumes/kubernetes.io~projected/kube-api-access-fbshl DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volumes/kubernetes.io~projected/kube-api-access-prn7r DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/96cf9d61-5fb4-45ed-b68b-868a6738f12c/volumes/kubernetes.io~projected/kube-api-access-nzqjq DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d0b2a03-5d09-4c44-bec0-ec66bdea6781/volumes/kubernetes.io~projected/kube-api-access-8fd66 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f88a9388-d3c1-421a-bb58-87a481f7333a/volumes/kubernetes.io~projected/kube-api-access-mtk22 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71eca11c-bf4c-473c-b64a-64041bbe808d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:377 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9192aec-64e7-4d0f-8066-e236dc75b87d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:415 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2a24e0c9-e7a1-4206-bdb5-a956b226a79e/volumes/kubernetes.io~projected/kube-api-access-94g2q DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/863826145f66d91883bb1609c0c8a89a873b6f661d014dd8dc326d87cea21c3e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3374c66fc7267c1 MacAddress:be:8e:87:d1:4a:ad Speed:10000 Mtu:8901} {Name:3b6338c4814c92b MacAddress:46:54:2e:6a:76:59 Speed:10000 Mtu:8901} {Name:42d8db269e2fd9f MacAddress:6e:cb:8f:74:fa:3c Speed:10000 Mtu:8901} {Name:8d1ffca31f6f8bd MacAddress:ca:31:74:09:7d:38 Speed:10000 Mtu:8901} {Name:a259acccbe5603c MacAddress:a6:a2:ef:94:3b:b9 Speed:10000 Mtu:8901} {Name:ad9e327416fb0db MacAddress:6e:76:a7:c5:35:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:db:99:e3:c1 Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:95:13:08:6e:bd Speed:0 Mtu:8901} {Name:e02ba45412bcdc8 MacAddress:72:b9:e5:20:34:9f Speed:10000 Mtu:8901} {Name:e65f1d2471da591 MacAddress:de:c7:0b:1e:cc:71 Speed:10000 Mtu:8901} {Name:e92b562f3610f65 MacAddress:6a:25:a5:a1:9e:0e Speed:10000 Mtu:8901} {Name:edd42781e655509 MacAddress:4e:92:71:e6:e5:c9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:db:99:e3:c1 Speed:0 Mtu:9001} {Name:fa00c09ca1049ac MacAddress:de:7f:6d:60:33:9a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0a:c3:a1:2f:ed:03 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:fa:d9:60:51:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959064    6203 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959118    6203 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959407    6203 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959438    6203 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959572    6203 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959579    6203 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959590    6203 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959603    6203 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 23:13:41.959753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959715    6203 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 23:13:41.960253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.959897    6203 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 23:13:41.960253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.960009    6203 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 23:13:41.960253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.960045    6203 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 23:13:41.960253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.960055    6203 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 23:13:41.960253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.960065    6203 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 23:13:41.960253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.960074    6203 kubelet.go:405] "Adding apiserver pod source"
Oct 29 23:13:41.960253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.960090    6203 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 23:13:41.961016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.960999    6203 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 23:13:41.961016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.961013    6203 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 23:13:41.961594 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.961575    6203 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 23:13:41.965016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965001    6203 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 23:13:41.965016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965015    6203 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 23:13:41.965139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965129    6203 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 23:13:41.965354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965343    6203 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 23:13:41.965393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965373    6203 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 23:13:41.965485 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965471    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 23:13:41.965485 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965487    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965493    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965499    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965508    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965518    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965525    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965531    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965540    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965545    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965555    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965564    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965589    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 23:13:41.965592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.965595    6203 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 23:13:41.969182 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.969167    6203 server.go:1263] "Started kubelet"
Oct 29 23:13:41.969292 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.969248    6203 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 23:13:41.969696 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.969628    6203 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 23:13:41.969807 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.969723    6203 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 23:13:41.969749 ip-10-0-139-224 systemd[1]: Started Kubernetes Kubelet.
Oct 29 23:13:41.970095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.970067    6203 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 23:13:41.971271 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.971252    6203 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 23:13:41.975309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.975289    6203 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 23:13:41.975434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.975315    6203 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 23:13:41.975434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.975428    6203 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 23:13:41.975434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.975439    6203 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 23:13:41.975671 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:41.975466    6203 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-224.ec2.internal\" not found"
Oct 29 23:13:41.975671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.975526    6203 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 23:13:41.976708 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.975393    6203 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 23:07:33 +0000 UTC" deadline="2028-06-12 01:31:53.824651585 +0000 UTC"
Oct 29 23:13:41.976820 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.976713    6203 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14186h18m11.847946434s"
Oct 29 23:13:41.977833 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.977814    6203 factory.go:55] Registering systemd factory
Oct 29 23:13:41.977833 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.977836    6203 factory.go:223] Registration of the systemd container factory successfully
Oct 29 23:13:41.978233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978216    6203 factory.go:153] Registering CRI-O factory
Oct 29 23:13:41.978233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978234    6203 factory.go:223] Registration of the crio container factory successfully
Oct 29 23:13:41.978361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978290    6203 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 23:13:41.978361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978308    6203 factory.go:103] Registering Raw factory
Oct 29 23:13:41.978361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978320    6203 manager.go:1199] Started watching for new ooms in manager
Oct 29 23:13:41.978361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978302    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db" seLinuxMountContext=""
Oct 29 23:13:41.978361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978339    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 29 23:13:41.978361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978354    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d0b2a03-5d09-4c44-bec0-ec66bdea6781" volumeName="kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978369    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21b1ae7e-9d23-46d1-ada5-a0e6b66c1418" volumeName="kubernetes.io/projected/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-kube-api-access-prn7r" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978383    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9685c622-c264-4e53-8542-7a07650a6e84" volumeName="kubernetes.io/projected/9685c622-c264-4e53-8542-7a07650a6e84-kube-api-access-vgrh6" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978397    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96cf9d61-5fb4-45ed-b68b-868a6738f12c" volumeName="kubernetes.io/configmap/96cf9d61-5fb4-45ed-b68b-868a6738f12c-serviceca" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978409    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96cf9d61-5fb4-45ed-b68b-868a6738f12c" volumeName="kubernetes.io/projected/96cf9d61-5fb4-45ed-b68b-868a6738f12c-kube-api-access-nzqjq" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978421    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978435    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21b1ae7e-9d23-46d1-ada5-a0e6b66c1418" volumeName="kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-env-overrides" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978447    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9685c622-c264-4e53-8542-7a07650a6e84" volumeName="kubernetes.io/empty-dir/9685c622-c264-4e53-8542-7a07650a6e84-data-volume" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978459    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9a2e31a-049d-48d2-979c-8557f7c239f2" volumeName="kubernetes.io/projected/d9a2e31a-049d-48d2-979c-8557f7c239f2-kube-api-access-dfxp2" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978474    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978486    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978498    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71eca11c-bf4c-473c-b64a-64041bbe808d" volumeName="kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-metrics-client-ca" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978518    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ec3c2-c0d3-4312-ad94-16b6b8ba6067" volumeName="kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-tmp" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978530    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ec3c2-c0d3-4312-ad94-16b6b8ba6067" volumeName="kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-tuned" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978543    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1769c496-3b59-4aba-a674-13dae505ed09" volumeName="kubernetes.io/configmap/1769c496-3b59-4aba-a674-13dae505ed09-config-volume" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978557    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978574    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978588    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978604    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f" volumeName="kubernetes.io/empty-dir/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-volume-directive-shadow" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978620    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f643dab0-17a4-410b-b56b-f278862fa237" volumeName="kubernetes.io/projected/f643dab0-17a4-410b-b56b-f278862fa237-kube-api-access-lpflj" seLinuxMountContext=""
Oct 29 23:13:41.978640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978636    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1769c496-3b59-4aba-a674-13dae505ed09" volumeName="kubernetes.io/empty-dir/1769c496-3b59-4aba-a674-13dae505ed09-tmp-dir" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978653    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978669    6203 manager.go:319] Starting recovery of all containers
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978669    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21b1ae7e-9d23-46d1-ada5-a0e6b66c1418" volumeName="kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978687    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71eca11c-bf4c-473c-b64a-64041bbe808d" volumeName="kubernetes.io/projected/71eca11c-bf4c-473c-b64a-64041bbe808d-kube-api-access-vw22s" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978703    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/empty-dir/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-tmp" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978721    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-grpc-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978759    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9685c622-c264-4e53-8542-7a07650a6e84" volumeName="kubernetes.io/configmap/9685c622-c264-4e53-8542-7a07650a6e84-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978800    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1769c496-3b59-4aba-a674-13dae505ed09" volumeName="kubernetes.io/projected/1769c496-3b59-4aba-a674-13dae505ed09-kube-api-access-5k7k8" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978818    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1769c496-3b59-4aba-a674-13dae505ed09" volumeName="kubernetes.io/secret/1769c496-3b59-4aba-a674-13dae505ed09-metrics-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978836    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d0b2a03-5d09-4c44-bec0-ec66bdea6781" volumeName="kubernetes.io/projected/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-kube-api-access-8fd66" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978859    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71eca11c-bf4c-473c-b64a-64041bbe808d" volumeName="kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978889    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71eca11c-bf4c-473c-b64a-64041bbe808d" volumeName="kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978903    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53" volumeName="kubernetes.io/projected/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-kube-api-access-r88bf" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978920    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978935    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e327984-d0fc-4330-a933-f104718bce7c" volumeName="kubernetes.io/projected/2e327984-d0fc-4330-a933-f104718bce7c-kube-api-access-tqr78" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978947    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978961    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53" volumeName="kubernetes.io/configmap/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-metrics-client-ca" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978973    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d735fb1d-a5b9-4d8d-af87-bdaded8f82e6" volumeName="kubernetes.io/projected/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-kube-api-access-r8qnv" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978986    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.978999    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979011    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979022    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e327984-d0fc-4330-a933-f104718bce7c" volumeName="kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979035    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="534d0681-f061-4879-9e5e-f734eebfc7a4" volumeName="kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979046    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e327984-d0fc-4330-a933-f104718bce7c" volumeName="kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-binary-copy" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979059    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f" volumeName="kubernetes.io/projected/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-api-access-fbshl" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979074    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/projected/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-kube-api-access-zm8cb" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979088    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f643dab0-17a4-410b-b56b-f278862fa237" volumeName="kubernetes.io/configmap/f643dab0-17a4-410b-b56b-f278862fa237-iptables-alerter-script" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979100    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faf7f273-a82a-40a5-811c-efb7281d1db6" volumeName="kubernetes.io/configmap/faf7f273-a82a-40a5-811c-efb7281d1db6-konnectivity-ca" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979111    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21b1ae7e-9d23-46d1-ada5-a0e6b66c1418" volumeName="kubernetes.io/secret/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979123    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979135    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53" volumeName="kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979149    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979162    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/configmap/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-metrics-client-ca" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979174    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d735fb1d-a5b9-4d8d-af87-bdaded8f82e6" volumeName="kubernetes.io/secret/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-monitoring-plugin-cert" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979202    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979215    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f" volumeName="kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-metrics-client-ca" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979227    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f" volumeName="kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979240    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9685c622-c264-4e53-8542-7a07650a6e84" volumeName="kubernetes.io/secret/9685c622-c264-4e53-8542-7a07650a6e84-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979255    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ec3c2-c0d3-4312-ad94-16b6b8ba6067" volumeName="kubernetes.io/projected/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-kube-api-access-rgbtb" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979266    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f88a9388-d3c1-421a-bb58-87a481f7333a" volumeName="kubernetes.io/projected/f88a9388-d3c1-421a-bb58-87a481f7333a-kube-api-access-mtk22" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979280    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979293    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a24e0c9-e7a1-4206-bdb5-a956b226a79e" volumeName="kubernetes.io/projected/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kube-api-access-94g2q" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979306    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71eca11c-bf4c-473c-b64a-64041bbe808d" volumeName="kubernetes.io/empty-dir/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-textfile" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979321    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979333    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f88a9388-d3c1-421a-bb58-87a481f7333a" volumeName="kubernetes.io/secret/f88a9388-d3c1-421a-bb58-87a481f7333a-cert" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979348    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="faf7f273-a82a-40a5-811c-efb7281d1db6" volumeName="kubernetes.io/secret/faf7f273-a82a-40a5-811c-efb7281d1db6-agent-certs" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979360    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979371    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e327984-d0fc-4330-a933-f104718bce7c" volumeName="kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979383    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f" volumeName="kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979396    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71eca11c-bf4c-473c-b64a-64041bbe808d" volumeName="kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979408    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53" volumeName="kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979418    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9192aec-64e7-4d0f-8066-e236dc75b87d" volumeName="kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979444    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9a2e31a-049d-48d2-979c-8557f7c239f2" volumeName="kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-cni-binary-copy" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979456    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9a2e31a-049d-48d2-979c-8557f7c239f2" volumeName="kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-daemon-config" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979467    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3" volumeName="kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979481    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979493    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21b1ae7e-9d23-46d1-ada5-a0e6b66c1418" volumeName="kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979504    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979522    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979534    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f" volumeName="kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-tls" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979547    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89e02182-3b19-41c8-9040-f75a4254187b" volumeName="kubernetes.io/empty-dir/89e02182-3b19-41c8-9040-f75a4254187b-tmp-dir" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979559    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89e02182-3b19-41c8-9040-f75a4254187b" volumeName="kubernetes.io/projected/89e02182-3b19-41c8-9040-f75a4254187b-kube-api-access-kgcz9" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979576    6203 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18375e9b-7089-4b3a-b691-bd0ba7a3441d" volumeName="kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979601    6203 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 23:13:41.979591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:41.979610    6203 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 23:13:42.009286 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.009249    6203 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 23:13:42.010674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.010642    6203 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 23:13:42.010674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.010671    6203 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 23:13:42.010817 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.010696    6203 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 23:13:42.010817 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:42.010739    6203 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 23:13:42.028634 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.028603    6203 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 23:13:42.031931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.031893    6203 generic.go:356] "Generic (PLEG): container finished" podID="410469b0f63ac5a751ec52b1c6fe6043" containerID="04b1d4d4e30774d3a14a505b09b87f0bda92441f0049a381ead4362ef389f5f4" exitCode=0
Oct 29 23:13:42.036221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.036187    6203 generic.go:356] "Generic (PLEG): container finished" podID="71eca11c-bf4c-473c-b64a-64041bbe808d" containerID="ac55a5c5280e689e670398bba910f9b2192f8599bd36b2148ea69cf935331e37" exitCode=0
Oct 29 23:13:42.043322 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.043285    6203 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:13:42.054093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.054054    6203 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:13:42.054242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.054177    6203 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:13:42.054242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.054229    6203 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 23:13:42.062021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.061980    6203 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="42e6348341b119e421c41f88183fdeaf7597925a3800a0555ecd6d94caea888f" exitCode=0
Oct 29 23:13:42.071786 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.071743    6203 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="adecc820fc452016731415d37d6bcafab7fa51688a3de2fa7487ea9e89cdfb0f" exitCode=0
Oct 29 23:13:42.071786 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.071779    6203 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="92e5f0680859cfdc12a2f1e02017b6a90b37babedfcc024813051c21e59547ce" exitCode=0
Oct 29 23:13:42.071786 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.071789    6203 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="76ef1da2368ec878f99f9d42047532f738cb480daf4bd905e7762a32c679818b" exitCode=0
Oct 29 23:13:42.072060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.071802    6203 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="a292bc58cf2bd00353c5b034c73feb69866c20ba9a0d0dc574bc58e8459b4b7e" exitCode=0
Oct 29 23:13:42.072060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.071812    6203 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="d9abae1b4a08887c3f435fc97992e624e8a763b7187277243d6bf87a9f1ae623" exitCode=0
Oct 29 23:13:42.072060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.071824    6203 generic.go:356] "Generic (PLEG): container finished" podID="2e327984-d0fc-4330-a933-f104718bce7c" containerID="63254f042f1734236d442580f7475c7a9634f92520b660f4a9e7ada0d620f143" exitCode=0
Oct 29 23:13:42.087757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.087688    6203 manager.go:324] Recovery completed
Oct 29 23:13:42.112488 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:42.112429    6203 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 23:13:42.113467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113449    6203 cpu_manager.go:226] "Starting" policy="none"
Oct 29 23:13:42.113520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113467    6203 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 23:13:42.113520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113495    6203 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 23:13:42.113669 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113655    6203 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 23:13:42.113707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113669    6203 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 23:13:42.113707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113688    6203 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 23:13:42.113707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113693    6203 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 23:13:42.113707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113698    6203 policy_none.go:50] "Start"
Oct 29 23:13:42.113707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113707    6203 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 23:13:42.113850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113717    6203 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 23:13:42.113850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113847    6203 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 23:13:42.113917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113854    6203 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 23:13:42.113917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.113859    6203 policy_none.go:44] "Start"
Oct 29 23:13:42.118535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.118490    6203 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 23:13:42.121264 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.121249    6203 manager.go:341] "Starting Device Plugin manager"
Oct 29 23:13:42.121755 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:42.121281    6203 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 23:13:42.121808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.121757    6203 server.go:87] "Starting device plugin registration server"
Oct 29 23:13:42.122071 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.122060    6203 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 23:13:42.122103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.122072    6203 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 23:13:42.122192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.122172    6203 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 23:13:42.122319 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.122291    6203 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 23:13:42.122387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.122334    6203 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 23:13:42.122387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.122342    6203 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 23:13:42.122482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.122408    6203 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T23:13:42.122316957Z","UUID":"7bbd66c4-6e64-4369-9069-4820d948b121","Handler":null,"Name":"","Endpoint":""}
Oct 29 23:13:42.123894 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.123856    6203 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 23:13:42.123983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.123901    6203 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 23:13:42.124482 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:42.124258    6203 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 23:13:42.222761 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.222714    6203 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 23:13:42.224955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.224936    6203 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 23:13:42.225047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.224969    6203 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 23:13:42.225047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.224981    6203 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-224.ec2.internal" event="NodeHasSufficientPID"
Oct 29 23:13:42.225047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.225028    6203 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.265873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.265838    6203 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.266054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.265972    6203 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.313386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.313327    6203 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"]
Oct 29 23:13:42.314045 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.314006    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal" event={"ID":"410469b0f63ac5a751ec52b1c6fe6043","Type":"ContainerStarted","Data":"b475dfa91cb8980648c73f2e9aa5c8c9f888d86e3556a942ab30f92662451ff6"}
Oct 29 23:13:42.314156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.314055    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal" event={"ID":"410469b0f63ac5a751ec52b1c6fe6043","Type":"ContainerDied","Data":"04b1d4d4e30774d3a14a505b09b87f0bda92441f0049a381ead4362ef389f5f4"}
Oct 29 23:13:42.314156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.314079    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal" event={"ID":"410469b0f63ac5a751ec52b1c6fe6043","Type":"ContainerStarted","Data":"80ba10f609dd3fc792bc1637a4ab2c7ef6a1f740a245ebf73b52f700fed28696"}
Oct 29 23:13:42.314156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.314124    6203 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.314156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.314145    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal" event={"ID":"88f04efcc82b4a40de30b52c3590d78c","Type":"ContainerStarted","Data":"30d989fe00db8aa2d6d8f757f952a13a5501247158e5caeab16a05dae65f6483"}
Oct 29 23:13:42.314325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.314165    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal" event={"ID":"88f04efcc82b4a40de30b52c3590d78c","Type":"ContainerStarted","Data":"3bfeaca961dc8948cbc07462a91d48a8f4e0447369dd9f8805aa4a891882cd7e"}
Oct 29 23:13:42.314370 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.314330    6203 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.339841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.339760    6203 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 23:13:42.339841 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:42.339825    6203 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.340514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.340497    6203 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 23:13:42.340567 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:42.340539    6203 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.429901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.429839    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88f04efcc82b4a40de30b52c3590d78c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" (UID: \"88f04efcc82b4a40de30b52c3590d78c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.429901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.429897    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.430053 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.429916    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.530505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.530459    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.530713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.530596    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.530713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.530628    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.530713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.530669    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/410469b0f63ac5a751ec52b1c6fe6043-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" (UID: \"410469b0f63ac5a751ec52b1c6fe6043\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.530713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.530673    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88f04efcc82b4a40de30b52c3590d78c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" (UID: \"88f04efcc82b4a40de30b52c3590d78c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.530847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.530728    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/88f04efcc82b4a40de30b52c3590d78c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" (UID: \"88f04efcc82b4a40de30b52c3590d78c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:42.961266 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:42.961222    6203 apiserver.go:52] "Watching apiserver"
Oct 29 23:13:43.019062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.019031    6203 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 23:13:43.019632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.019611    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-mzt69","openshift-insights/insights-runtime-extractor-gkt7z","openshift-monitoring/thanos-querier-866cccd46d-46pmg","openshift-network-diagnostics/network-check-target-b97hf","openshift-network-operator/iptables-alerter-glknz","openshift-multus/multus-2kg5q","openshift-ovn-kubernetes/ovnkube-node-dv7ws","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn","openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m","openshift-monitoring/node-exporter-f24wt","kube-system/konnectivity-agent-44xj8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal","openshift-dns/dns-default-kjc8k","openshift-image-registry/node-ca-qwc9k","openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg","openshift-cluster-node-tuning-operator/tuned-4bvzv","openshift-dns/node-resolver-zx5vd","kube-system/global-pull-secret-syncer-mkj2j","openshift-ingress-canary/ingress-canary-mfwdq","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-bc5z8","kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"]
Oct 29 23:13:43.037098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.037072    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 23:13:43.037493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.037477    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 23:13:43.037493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.037482    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5wvkd\""
Oct 29 23:13:43.040016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.040000    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 23:13:43.040228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.040216    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.040345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.040321    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 23:13:43.040454 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.040378    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 23:13:43.040497 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.040466    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.042047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042026    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.042047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042038    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.042283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042268    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.042369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042356    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.042488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042475    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.042618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042601    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.042785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042770    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 23:13:43.042824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042792    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wscf4\""
Oct 29 23:13:43.042824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042794    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m6t5c\""
Oct 29 23:13:43.042900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042822    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fjkb\""
Oct 29 23:13:43.042900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042828    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 23:13:43.042900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042848    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 23:13:43.042990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042934    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vsvpd\""
Oct 29 23:13:43.042990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042964    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 23:13:43.042990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.042966    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfk9z\""
Oct 29 23:13:43.043090 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.043001    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sw5tb\""
Oct 29 23:13:43.043357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.043344    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.058129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058104    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.058129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058121    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058126    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058154    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058177    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058181    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058188    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058200    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058188    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 23:13:43.058277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.058263    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 23:13:43.059767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.059753    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.060282 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.060270    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.060384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.060371    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.060655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.060643    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.060707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.060695    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 23:13:43.060807 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.060796    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.064074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.064061    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 23:13:43.064177 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.064126    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.064231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.064217    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.064332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.064318    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 23:13:43.064386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.064318    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 23:13:43.064386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.064379    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 23:13:43.064931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.064919    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 23:13:43.065271 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.065246    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 23:13:43.065271 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.065248    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 29 23:13:43.071850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.071819    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 29 23:13:43.071950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.071894    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 23:13:43.071950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.071917    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zlts4\""
Oct 29 23:13:43.071950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.071925    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 29 23:13:43.071950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.071935    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 29 23:13:43.071950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.071940    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 23:13:43.072206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072049    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6nojl5jppa58c\""
Oct 29 23:13:43.072206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072109    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 29 23:13:43.072206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072121    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 23:13:43.072206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072185    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 23:13:43.072206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072204    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 23:13:43.072393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072213    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 23:13:43.072393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072273    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 29 23:13:43.072393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072299    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xvwxf\""
Oct 29 23:13:43.072393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072320    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 23:13:43.072393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.072379    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w5p5k\""
Oct 29 23:13:43.074610 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.074591    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 23:13:43.074728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.074645    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 29 23:13:43.074728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.074651    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 29 23:13:43.074728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.074725    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8hh8s\""
Oct 29 23:13:43.074893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.074707    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3da6io525po6t\""
Oct 29 23:13:43.074893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.074695    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5hpb5\""
Oct 29 23:13:43.075282 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.075269    6203 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:43.075372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.075360    6203 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:43.077577 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.077562    6203 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 23:13:43.079238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079223    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bk87z\""
Oct 29 23:13:43.079314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079299    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 23:13:43.079471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079453    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 23:13:43.079471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079467    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079471    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96xtz\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079514    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kcr8p\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079521    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079547    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079554    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079561    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079571    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079584    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079590    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z28mk\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079611    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rxpzs\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079619    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079614    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079622    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cxqm7\""
Oct 29 23:13:43.079640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079646    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 23:13:43.080036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079613    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 23:13:43.080036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079727    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88spp\""
Oct 29 23:13:43.080036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079776    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rlkhk\""
Oct 29 23:13:43.080036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079798    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h2str\""
Oct 29 23:13:43.080036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.079826    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 29 23:13:43.100687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.100673    6203 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 23:13:43.100757 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:43.100715    6203 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-224.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:43.101180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.101155    6203 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 23:13:43.101269 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:13:43.101191    6203 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-224.ec2.internal"
Oct 29 23:13:43.133897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.133851    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.134041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.133908    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-cnibin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.134041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.133934    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-var-lib-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.134041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.133955    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.134041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.133980    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-lib-modules\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.134041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134002    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-systemd\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.134230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134057    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbtb\" (UniqueName: \"kubernetes.io/projected/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-kube-api-access-rgbtb\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.134230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134104    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.134230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134137    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.134230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134169    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-kubelet\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.134230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134183    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.134230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134194    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-multus-certs\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.134492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134264    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-tuned\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.134492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134271    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.134492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134328    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.134492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134358    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-tuned\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.134492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134408    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-netns\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.134492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134471    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134507    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f643dab0-17a4-410b-b56b-f278862fa237-iptables-alerter-script\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134541    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcz9\" (UniqueName: \"kubernetes.io/projected/89e02182-3b19-41c8-9040-f75a4254187b-kube-api-access-kgcz9\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134568    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134642    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134660    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-tls\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134679    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96cf9d61-5fb4-45ed-b68b-868a6738f12c-serviceca\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134698    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88a9388-d3c1-421a-bb58-87a481f7333a-cert\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134697    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f643dab0-17a4-410b-b56b-f278862fa237-iptables-alerter-script\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134800    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fd66\" (UniqueName: \"kubernetes.io/projected/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-kube-api-access-8fd66\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:43.134825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134818    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134839    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tms9l\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134901    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134917    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88a9388-d3c1-421a-bb58-87a481f7333a-cert\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134922    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-tls\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134940    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovn-node-metrics-cert\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134926    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-cni-binary-copy\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.134982    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-etc-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135011    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-env-overrides\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135057    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94g2q\" (UniqueName: \"kubernetes.io/projected/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kube-api-access-94g2q\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135089    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-dbus\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135119    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9685c622-c264-4e53-8542-7a07650a6e84-data-volume\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135139    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-metrics-certs\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135147    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1769c496-3b59-4aba-a674-13dae505ed09-config-volume\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135058    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135155    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-cni-binary-copy\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135171    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-env-overrides\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135186    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-metrics-client-ca\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135215    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9685c622-c264-4e53-8542-7a07650a6e84-data-volume\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.135294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135245    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-sys\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135338    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-tmp\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135356    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1769c496-3b59-4aba-a674-13dae505ed09-config-volume\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135376    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf7f273-a82a-40a5-811c-efb7281d1db6-konnectivity-ca\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135395    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-metrics-client-ca\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135400    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135435    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1769c496-3b59-4aba-a674-13dae505ed09-metrics-tls\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135462    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7k8\" (UniqueName: \"kubernetes.io/projected/1769c496-3b59-4aba-a674-13dae505ed09-kube-api-access-5k7k8\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135466    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/96cf9d61-5fb4-45ed-b68b-868a6738f12c-serviceca\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135486    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-sys\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135509    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-registration-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135434    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-tmp\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135550    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpflj\" (UniqueName: \"kubernetes.io/projected/f643dab0-17a4-410b-b56b-f278862fa237-kube-api-access-lpflj\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135553    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/faf7f273-a82a-40a5-811c-efb7281d1db6-konnectivity-ca\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135587    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89e02182-3b19-41c8-9040-f75a4254187b-hosts-file\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135614    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1769c496-3b59-4aba-a674-13dae505ed09-metrics-tls\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135618    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqr78\" (UniqueName: \"kubernetes.io/projected/2e327984-d0fc-4330-a933-f104718bce7c-kube-api-access-tqr78\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135623    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135730    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88bf\" (UniqueName: \"kubernetes.io/projected/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-kube-api-access-r88bf\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135776    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135869    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135934    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9685c622-c264-4e53-8542-7a07650a6e84-crio-socket\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.135989    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-system-cni-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136031    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136044    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136063    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136091    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.136106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136109    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136144    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136162    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prn7r\" (UniqueName: \"kubernetes.io/projected/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-kube-api-access-prn7r\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136177    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-host\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136205    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136293    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136337    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136367    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136367    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136439    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgrh6\" (UniqueName: \"kubernetes.io/projected/9685c622-c264-4e53-8542-7a07650a6e84-kube-api-access-vgrh6\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136422    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-accelerators-collector-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136558    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136584    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136605    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-bin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136650    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136667    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136686    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9192aec-64e7-4d0f-8066-e236dc75b87d-original-pull-secret\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136712    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136717    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-root\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136769    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136811    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136831    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-systemd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136842    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136845    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-conf\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136907    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-metrics-client-ca\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.136945    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-k8s-cni-cncf-io\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137028    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137060    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9685c622-c264-4e53-8542-7a07650a6e84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137076    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-metrics-client-ca\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137095    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw22s\" (UniqueName: \"kubernetes.io/projected/71eca11c-bf4c-473c-b64a-64041bbe808d-kube-api-access-vw22s\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137122    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-bin\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137152    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137181    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137224    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-var-lib-kubelet\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.137229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137248    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysconfig\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137287    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9685c622-c264-4e53-8542-7a07650a6e84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137290    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137347    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137362    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137377    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137396    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e327984-d0fc-4330-a933-f104718bce7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137406    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137436    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxp2\" (UniqueName: \"kubernetes.io/projected/d9a2e31a-049d-48d2-979c-8557f7c239f2-kube-api-access-dfxp2\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137464    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-modprobe-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137488    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-kubernetes\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137520    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137542    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf7f273-a82a-40a5-811c-efb7281d1db6-agent-certs\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137559    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137574    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-tmp\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137611    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137640    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137644    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-tmp\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137666    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137694    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137721    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-systemd-units\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137750    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-netns\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137779    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-log-socket\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137808    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-socket-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137723    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137817    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/faf7f273-a82a-40a5-811c-efb7281d1db6-agent-certs\") pod \"konnectivity-agent-44xj8\" (UID: \"faf7f273-a82a-40a5-811c-efb7281d1db6\") " pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137838    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137812    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137910    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-hostroot\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137914    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137940    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.137986    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1769c496-3b59-4aba-a674-13dae505ed09-tmp-dir\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138018    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-daemon-config\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138063    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138074    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f643dab0-17a4-410b-b56b-f278862fa237-host-slash\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138112    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138129    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1769c496-3b59-4aba-a674-13dae505ed09-tmp-dir\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138147    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138166    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138256    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-wtmp\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138287    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cf9d61-5fb4-45ed-b68b-868a6738f12c-host\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138309    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-device-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138312    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138308    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-daemon-config\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138354    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-etc-kubernetes\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138391    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-kubelet-config\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138414    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-script-lib\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138461    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138495    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138541    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-multus\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138566    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-script-lib\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138573    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138591    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-ovn\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138629    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138631    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbshl\" (UniqueName: \"kubernetes.io/projected/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-api-access-fbshl\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138661    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-system-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138691    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-socket-dir-parent\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138708    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-kubelet\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138728    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.138766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138764    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e02182-3b19-41c8-9040-f75a4254187b-tmp-dir\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138860    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-config\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138861    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138893    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89e02182-3b19-41c8-9040-f75a4254187b-tmp-dir\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138900    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-cnibin\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.138980    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-textfile\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139004    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139021    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139090    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139102    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-ovnkube-config\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139170    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-textfile\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139145    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qnv\" (UniqueName: \"kubernetes.io/projected/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-kube-api-access-r8qnv\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139183    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139227    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-run\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139222    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139290    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqjq\" (UniqueName: \"kubernetes.io/projected/96cf9d61-5fb4-45ed-b68b-868a6738f12c-kube-api-access-nzqjq\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139293    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139320    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtk22\" (UniqueName: \"kubernetes.io/projected/f88a9388-d3c1-421a-bb58-87a481f7333a-kube-api-access-mtk22\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139420    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139441    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-grpc-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139456    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8cb\" (UniqueName: \"kubernetes.io/projected/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-kube-api-access-zm8cb\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139497    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139530    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-conf-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139557    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-sys-fs\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139583    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-node-log\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139615    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139622    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139636    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139655    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9685c622-c264-4e53-8542-7a07650a6e84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139687    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139721    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139824    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9685c622-c264-4e53-8542-7a07650a6e84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139825    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-slash\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139825    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-grpc-tls\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139862    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-netd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139917    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139922    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.139943    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.140032    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.140040    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.140181    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.140868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.140251    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.160840 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.160814    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcz9\" (UniqueName: \"kubernetes.io/projected/89e02182-3b19-41c8-9040-f75a4254187b-kube-api-access-kgcz9\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:13:43.160840 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.160826    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94g2q\" (UniqueName: \"kubernetes.io/projected/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kube-api-access-94g2q\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.161240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.161219    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbtb\" (UniqueName: \"kubernetes.io/projected/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-kube-api-access-rgbtb\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.161434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.161417    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tms9l\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\") pod \"prometheus-k8s-0\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:43.161633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.161617    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fd66\" (UniqueName: \"kubernetes.io/projected/1d0b2a03-5d09-4c44-bec0-ec66bdea6781-kube-api-access-8fd66\") pod \"network-metrics-daemon-bc5z8\" (UID: \"1d0b2a03-5d09-4c44-bec0-ec66bdea6781\") " pod="openshift-multus/network-metrics-daemon-bc5z8"
Oct 29 23:13:43.167873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.167847    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxp2\" (UniqueName: \"kubernetes.io/projected/d9a2e31a-049d-48d2-979c-8557f7c239f2-kube-api-access-dfxp2\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.168072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.168043    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7k8\" (UniqueName: \"kubernetes.io/projected/1769c496-3b59-4aba-a674-13dae505ed09-kube-api-access-5k7k8\") pod \"dns-default-kjc8k\" (UID: \"1769c496-3b59-4aba-a674-13dae505ed09\") " pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:43.168194 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.168109    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw22s\" (UniqueName: \"kubernetes.io/projected/71eca11c-bf4c-473c-b64a-64041bbe808d-kube-api-access-vw22s\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.168194 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.168132    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpflj\" (UniqueName: \"kubernetes.io/projected/f643dab0-17a4-410b-b56b-f278862fa237-kube-api-access-lpflj\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:13:43.168305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.168280    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prn7r\" (UniqueName: \"kubernetes.io/projected/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-kube-api-access-prn7r\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.168362 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.168326    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88bf\" (UniqueName: \"kubernetes.io/projected/bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53-kube-api-access-r88bf\") pod \"openshift-state-metrics-7bd7489957-7mmsg\" (UID: \"bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7mmsg"
Oct 29 23:13:43.168534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.168512    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqr78\" (UniqueName: \"kubernetes.io/projected/2e327984-d0fc-4330-a933-f104718bce7c-kube-api-access-tqr78\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.173085 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.173063    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqjq\" (UniqueName: \"kubernetes.io/projected/96cf9d61-5fb4-45ed-b68b-868a6738f12c-kube-api-access-nzqjq\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:13:43.173085 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.173072    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgrh6\" (UniqueName: \"kubernetes.io/projected/9685c622-c264-4e53-8542-7a07650a6e84-kube-api-access-vgrh6\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.173726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.173704    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qnv\" (UniqueName: \"kubernetes.io/projected/d735fb1d-a5b9-4d8d-af87-bdaded8f82e6-kube-api-access-r8qnv\") pod \"monitoring-plugin-77f4f76594-t2sfs\" (UID: \"d735fb1d-a5b9-4d8d-af87-bdaded8f82e6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:43.173821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.173807    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbshl\" (UniqueName: \"kubernetes.io/projected/677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f-kube-api-access-fbshl\") pod \"kube-state-metrics-5dc8d7d89b-s657m\" (UID: \"677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-s657m"
Oct 29 23:13:43.174112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.174096    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtk22\" (UniqueName: \"kubernetes.io/projected/f88a9388-d3c1-421a-bb58-87a481f7333a-kube-api-access-mtk22\") pod \"ingress-canary-mfwdq\" (UID: \"f88a9388-d3c1-421a-bb58-87a481f7333a\") " pod="openshift-ingress-canary/ingress-canary-mfwdq"
Oct 29 23:13:43.175059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.175043    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26l5f\" (UniqueName: \"kubernetes.io/projected/534d0681-f061-4879-9e5e-f734eebfc7a4-kube-api-access-26l5f\") pod \"network-check-target-b97hf\" (UID: \"534d0681-f061-4879-9e5e-f734eebfc7a4\") " pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:43.178733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.178717    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8cb\" (UniqueName: \"kubernetes.io/projected/7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3-kube-api-access-zm8cb\") pod \"thanos-querier-866cccd46d-46pmg\" (UID: \"7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3\") " pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:43.241467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241387    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-root\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.241467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241421    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241439    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-systemd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241463    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-conf\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241490    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-k8s-cni-cncf-io\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241492    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-root\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241508    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241516    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-bin\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241523    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-systemd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241545    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-var-lib-kubelet\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241563    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysconfig\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241557    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-k8s-cni-cncf-io\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241579    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-bin\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241631    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysconfig\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241637    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-var-lib-kubelet\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241641    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241669    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-modprobe-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241685    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-kubernetes\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241699    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241703    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-systemd-units\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241647    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-conf\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241736    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-netns\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241746    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-kubernetes\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.241746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241757    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-log-socket\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241775    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-socket-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241780    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-netns\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241792    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241757    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-systemd-units\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241826    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-hostroot\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241809    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-modprobe-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241826    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-log-socket\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241854    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-socket-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241855    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f643dab0-17a4-410b-b56b-f278862fa237-host-slash\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241846    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241902    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-hostroot\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241939    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f643dab0-17a4-410b-b56b-f278862fa237-host-slash\") pod \"iptables-alerter-glknz\" (UID: \"f643dab0-17a4-410b-b56b-f278862fa237\") " pod="openshift-network-operator/iptables-alerter-glknz"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241955    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-wtmp\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.241987    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cf9d61-5fb4-45ed-b68b-868a6738f12c-host\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242001    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cf9d61-5fb4-45ed-b68b-868a6738f12c-host\") pod \"node-ca-qwc9k\" (UID: \"96cf9d61-5fb4-45ed-b68b-868a6738f12c\") " pod="openshift-image-registry/node-ca-qwc9k"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242014    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-device-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242037    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-etc-kubernetes\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242056    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-device-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242068    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-kubelet-config\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242040    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-node-exporter-wtmp\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242087    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-etc-kubernetes\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242096    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-multus\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242116    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-kubelet-config\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242117    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-multus\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242120    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-ovn\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242141    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-ovn\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242144    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-system-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242167    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-socket-dir-parent\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242191    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-kubelet\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242218    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-cnibin\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242231    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-system-cni-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242238    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-kubelet\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242246    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-run\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242249    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-socket-dir-parent\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242264    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-cnibin\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242275    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-conf-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242295    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-run\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242298    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-sys-fs\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242316    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-multus-conf-dir\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242321    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-node-log\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242348    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-slash\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242367    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-sys-fs\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242374    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-netd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242391    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-slash\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242390    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-node-log\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242405    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-cni-netd\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242406    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-cnibin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242432    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-var-lib-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242440    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-cnibin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242455    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242480    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-lib-modules\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242487    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-var-lib-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242504    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-systemd\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242510    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242530    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-kubelet\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242555    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-multus-certs\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242577    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-kubelet\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242586    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-systemd\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242557    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-lib-modules\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242596    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-netns\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242616    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-multus-certs\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242629    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-etc-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242656    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-dbus\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242674    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-run-netns\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242686    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-etc-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242707    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-sys\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.242672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242725    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9192aec-64e7-4d0f-8066-e236dc75b87d-dbus\") pod \"global-pull-secret-syncer-mkj2j\" (UID: \"d9192aec-64e7-4d0f-8066-e236dc75b87d\") " pod="kube-system/global-pull-secret-syncer-mkj2j"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242728    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-sys\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242755    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-registration-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242764    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-sys\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242758    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71eca11c-bf4c-473c-b64a-64041bbe808d-sys\") pod \"node-exporter-f24wt\" (UID: \"71eca11c-bf4c-473c-b64a-64041bbe808d\") " pod="openshift-monitoring/node-exporter-f24wt"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242790    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89e02182-3b19-41c8-9040-f75a4254187b-hosts-file\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242802    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a24e0c9-e7a1-4206-bdb5-a956b226a79e-registration-dir\") pod \"aws-ebs-csi-driver-node-jvxfn\" (UID: \"2a24e0c9-e7a1-4206-bdb5-a956b226a79e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jvxfn"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242818    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9685c622-c264-4e53-8542-7a07650a6e84-crio-socket\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242843    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-system-cni-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242890    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/89e02182-3b19-41c8-9040-f75a4254187b-hosts-file\") pod \"node-resolver-zx5vd\" (UID: \"89e02182-3b19-41c8-9040-f75a4254187b\") " pod="openshift-dns/node-resolver-zx5vd"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242868    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242928    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242942    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242905    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9685c622-c264-4e53-8542-7a07650a6e84-crio-socket\") pod \"insights-runtime-extractor-gkt7z\" (UID: \"9685c622-c264-4e53-8542-7a07650a6e84\") " pod="openshift-insights/insights-runtime-extractor-gkt7z"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242929    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e327984-d0fc-4330-a933-f104718bce7c-system-cni-dir\") pod \"multus-additional-cni-plugins-mzt69\" (UID: \"2e327984-d0fc-4330-a933-f104718bce7c\") " pod="openshift-multus/multus-additional-cni-plugins-mzt69"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242954    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242991    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-host-run-ovn-kubernetes\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.242989    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-host\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.243052    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.243054    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-host\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.243077    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-bin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.243020    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d28ec3c2-c0d3-4312-ad94-16b6b8ba6067-etc-sysctl-d\") pod \"tuned-4bvzv\" (UID: \"d28ec3c2-c0d3-4312-ad94-16b6b8ba6067\") " pod="openshift-cluster-node-tuning-operator/tuned-4bvzv"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.243072    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21b1ae7e-9d23-46d1-ada5-a0e6b66c1418-run-openvswitch\") pod \"ovnkube-node-dv7ws\" (UID: \"21b1ae7e-9d23-46d1-ada5-a0e6b66c1418\") " pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:43.244676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.243153    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9a2e31a-049d-48d2-979c-8557f7c239f2-host-var-lib-cni-bin\") pod \"multus-2kg5q\" (UID: \"d9a2e31a-049d-48d2-979c-8557f7c239f2\") " pod="openshift-multus/multus-2kg5q"
Oct 29 23:13:43.768944 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.768914    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-44xj8"
Oct 29 23:13:43.794726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:43.794699    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-t2sfs"
Oct 29 23:13:44.258111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:44.258080    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kjc8k"
Oct 29 23:13:44.381031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:44.380993    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:13:46.674512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:46.674486    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:13:46.953464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:46.953386    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-866cccd46d-46pmg"
Oct 29 23:13:52.442825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:52.442790    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:53.171700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:53.171672    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:13:53.171939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:13:53.171924    6203 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 23:14:03.012098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.012020    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:14:03.012560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.012384    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="prometheus" containerID="cri-o://26a6a38a1174c1c01a23621ad6894bcaa7e95dfd6f621937d3395dcb43a09818" gracePeriod=600
Oct 29 23:14:03.012560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.012432    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="thanos-sidecar" containerID="cri-o://29e7cf4aad4bbcc502aabc64824d72707be2a8b1061694ad11c963e7ee745c98" gracePeriod=600
Oct 29 23:14:03.012560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.012490    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-web" containerID="cri-o://6ec853fb7226f9f743c7a4a2f93ad6e3596d5b4c666b7af90826f3ad970b2255" gracePeriod=600
Oct 29 23:14:03.012560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.012492    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="config-reloader" containerID="cri-o://1d69a5f12faa5024bd37239518a665d184b6f4a398c7dbe95abda92cdcb53728" gracePeriod=600
Oct 29 23:14:03.012560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.012464    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy" containerID="cri-o://667f5df4ba4fb0fcb0a69b672ae4ab239f509d152b73431a04bd9e0055872e9e" gracePeriod=600
Oct 29 23:14:03.012560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.012510    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://90ca07a0667961a7947f9e82f8e94415ac6781ea44391df19e4f5cb198fc287c" gracePeriod=600
Oct 29 23:14:03.153842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153816    6203 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="90ca07a0667961a7947f9e82f8e94415ac6781ea44391df19e4f5cb198fc287c" exitCode=0
Oct 29 23:14:03.153842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153842    6203 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="667f5df4ba4fb0fcb0a69b672ae4ab239f509d152b73431a04bd9e0055872e9e" exitCode=0
Oct 29 23:14:03.154008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153850    6203 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="29e7cf4aad4bbcc502aabc64824d72707be2a8b1061694ad11c963e7ee745c98" exitCode=0
Oct 29 23:14:03.154008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153856    6203 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="1d69a5f12faa5024bd37239518a665d184b6f4a398c7dbe95abda92cdcb53728" exitCode=0
Oct 29 23:14:03.154008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153864    6203 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="26a6a38a1174c1c01a23621ad6894bcaa7e95dfd6f621937d3395dcb43a09818" exitCode=0
Oct 29 23:14:03.154008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153895    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"90ca07a0667961a7947f9e82f8e94415ac6781ea44391df19e4f5cb198fc287c"}
Oct 29 23:14:03.154008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153968    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"667f5df4ba4fb0fcb0a69b672ae4ab239f509d152b73431a04bd9e0055872e9e"}
Oct 29 23:14:03.154008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.153995    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"29e7cf4aad4bbcc502aabc64824d72707be2a8b1061694ad11c963e7ee745c98"}
Oct 29 23:14:03.154180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.154019    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"1d69a5f12faa5024bd37239518a665d184b6f4a398c7dbe95abda92cdcb53728"}
Oct 29 23:14:03.154180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:03.154042    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"26a6a38a1174c1c01a23621ad6894bcaa7e95dfd6f621937d3395dcb43a09818"}
Oct 29 23:14:04.160445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.160413    6203 generic.go:356] "Generic (PLEG): container finished" podID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerID="6ec853fb7226f9f743c7a4a2f93ad6e3596d5b4c666b7af90826f3ad970b2255" exitCode=0
Oct 29 23:14:04.160789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.160481    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"6ec853fb7226f9f743c7a4a2f93ad6e3596d5b4c666b7af90826f3ad970b2255"}
Oct 29 23:14:04.205184 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.205158    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dv7ws"
Oct 29 23:14:04.757946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.757925    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:04.807283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807251    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807289    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807307    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807328    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807364    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807399    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807428    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807460    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807488    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807525    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807548    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807579    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807609    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807635    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807673    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807703    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807710    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807751    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807799    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.807825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807830    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.808821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.807866    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\") pod \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\" (UID: \"18375e9b-7089-4b3a-b691-bd0ba7a3441d\") "
Oct 29 23:14:04.808821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.808092    6203 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.808821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.808153    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:04.809161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.809133    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:04.810386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.810354    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:04.810523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.810501    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:04.810590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.810527    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:04.810643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.810607    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.810754 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.810656    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:04.810754 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.810728    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.811111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.811086    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:04.813025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.812998    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:04.813126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.812998    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.813126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.813054    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:14:04.813322 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.813280    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.813423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.813349    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.813588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.813565    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.813644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.813618    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.813697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.813682    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.814032 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.814010    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "kube-api-access-tms9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:04.823627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.823601    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config" pod "18375e9b-7089-4b3a-b691-bd0ba7a3441d" (UID: "18375e9b-7089-4b3a-b691-bd0ba7a3441d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:04.909030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.908950    6203 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.908978    6203 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.908991    6203 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909006    6203 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-tls-assets\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909018    6203 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-config-out\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909031    6203 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-kube-rbac-proxy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909043    6203 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-web-config\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909055    6203 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-grpc-tls\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909066    6203 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909078    6203 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-db\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909090    6203 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909103    6203 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909118    6203 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-metrics-client-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909131    6203 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909144    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tms9l\" (UniqueName: \"kubernetes.io/projected/18375e9b-7089-4b3a-b691-bd0ba7a3441d-kube-api-access-tms9l\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909157    6203 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909168    6203 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18375e9b-7089-4b3a-b691-bd0ba7a3441d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909181    6203 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:04.909354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:04.909194    6203 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18375e9b-7089-4b3a-b691-bd0ba7a3441d-secret-metrics-client-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:14:05.167028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.167002    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18375e9b-7089-4b3a-b691-bd0ba7a3441d","Type":"ContainerDied","Data":"e65f1d2471da59138e13da5877bdac86b57d30d5bad6ede01c74454d07f1b0df"}
Oct 29 23:14:05.167445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.167046    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.167445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.167092    6203 scope.go:122] "RemoveContainer" containerID="90ca07a0667961a7947f9e82f8e94415ac6781ea44391df19e4f5cb198fc287c"
Oct 29 23:14:05.174958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.174671    6203 scope.go:122] "RemoveContainer" containerID="667f5df4ba4fb0fcb0a69b672ae4ab239f509d152b73431a04bd9e0055872e9e"
Oct 29 23:14:05.181600 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.181582    6203 scope.go:122] "RemoveContainer" containerID="6ec853fb7226f9f743c7a4a2f93ad6e3596d5b4c666b7af90826f3ad970b2255"
Oct 29 23:14:05.189106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.189087    6203 scope.go:122] "RemoveContainer" containerID="29e7cf4aad4bbcc502aabc64824d72707be2a8b1061694ad11c963e7ee745c98"
Oct 29 23:14:05.195599 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.195579    6203 scope.go:122] "RemoveContainer" containerID="1d69a5f12faa5024bd37239518a665d184b6f4a398c7dbe95abda92cdcb53728"
Oct 29 23:14:05.202143 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.202128    6203 scope.go:122] "RemoveContainer" containerID="26a6a38a1174c1c01a23621ad6894bcaa7e95dfd6f621937d3395dcb43a09818"
Oct 29 23:14:05.209277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.209256    6203 scope.go:122] "RemoveContainer" containerID="42e6348341b119e421c41f88183fdeaf7597925a3800a0555ecd6d94caea888f"
Oct 29 23:14:05.228648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.228625    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:14:05.238711 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.238675    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:14:05.274733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.274701    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:14:05.274982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.274969    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="thanos-sidecar"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.274983    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="thanos-sidecar"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.274995    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="config-reloader"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275000    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="config-reloader"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275006    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="prometheus"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275011    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="prometheus"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275021    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275025    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy"
Oct 29 23:14:05.275030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275031    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-web"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275037    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-web"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275057    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-thanos"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275061    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-thanos"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275069    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="init-config-reloader"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275074    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="init-config-reloader"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275112    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-thanos"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275118    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="init-config-reloader"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275125    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="thanos-sidecar"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275130    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="config-reloader"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275135    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="prometheus"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275142    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy"
Oct 29 23:14:05.275251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.275146    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" containerName="kube-rbac-proxy-web"
Oct 29 23:14:05.279041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.279022    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.297031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.297001    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 29 23:14:05.297261 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.297247    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 29 23:14:05.297328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.297291    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 29 23:14:05.297377 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.297361    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 29 23:14:05.297425 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.297392    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 29 23:14:05.310354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.310328    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:14:05.312081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312052    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312101    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8f43368-e2d8-4fca-8f89-e3319058b079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312213 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312171    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-config\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312214    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312268    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312318 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312303    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312424 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312407    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312447    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312519 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312475    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312533    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312572    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312596    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8f43368-e2d8-4fca-8f89-e3319058b079-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312617    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312651    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312687    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312740    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 29 23:14:05.312847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312752    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fbd\" (UniqueName: \"kubernetes.io/projected/f8f43368-e2d8-4fca-8f89-e3319058b079-kube-api-access-k2fbd\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312858    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312916    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.312975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312952    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 29 23:14:05.313065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.312951    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.313065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.313005    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 23:14:05.313065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.313021    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.313232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.313209    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 29 23:14:05.327167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.327143    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 29 23:14:05.327167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.327142    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 29 23:14:05.396724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.396694    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 29 23:14:05.396724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.396722    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 29 23:14:05.396964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.396698    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6nojl5jppa58c\""
Oct 29 23:14:05.396964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.396779    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 29 23:14:05.396964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.396798    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bk87z\""
Oct 29 23:14:05.396964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.396798    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 29 23:14:05.414337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414311    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414345    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8f43368-e2d8-4fca-8f89-e3319058b079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414362    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-config\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414477    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414503    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414521    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414541    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414588    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414620    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414643    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414672    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414698    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8f43368-e2d8-4fca-8f89-e3319058b079-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414721    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414748    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414769    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414793    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fbd\" (UniqueName: \"kubernetes.io/projected/f8f43368-e2d8-4fca-8f89-e3319058b079-kube-api-access-k2fbd\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.414855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.414793    6203 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.415325    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.415381    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.415394    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.415416    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.415440    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.415603    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.415801    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.416015    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.416092    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.416936    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.417702    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.417943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.417830    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.418097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.418069    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.418898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.418856    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8f43368-e2d8-4fca-8f89-e3319058b079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.419011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.418941    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-config\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.419597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.419571    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8f43368-e2d8-4fca-8f89-e3319058b079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.419695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.419625    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.420028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.420010    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.420489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.420464    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8f43368-e2d8-4fca-8f89-e3319058b079-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.420654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.420637    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.420730 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.420660    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.420788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.420726    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.421061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.421040    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8f43368-e2d8-4fca-8f89-e3319058b079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.439701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.439671    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fbd\" (UniqueName: \"kubernetes.io/projected/f8f43368-e2d8-4fca-8f89-e3319058b079-kube-api-access-k2fbd\") pod \"prometheus-k8s-0\" (UID: \"f8f43368-e2d8-4fca-8f89-e3319058b079\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.598641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.598596    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:05.931672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:05.931648    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 29 23:14:05.933571 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:14:05.933532    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f43368_e2d8_4fca_8f89_e3319058b079.slice/crio-720067f2027159b7b2c3111957de0e72008be5f5a67c6cec85dfcf9b8bf36a96 WatchSource:0}: Error finding container 720067f2027159b7b2c3111957de0e72008be5f5a67c6cec85dfcf9b8bf36a96: Status 404 returned error can't find the container with id 720067f2027159b7b2c3111957de0e72008be5f5a67c6cec85dfcf9b8bf36a96
Oct 29 23:14:06.016049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:06.016014    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18375e9b-7089-4b3a-b691-bd0ba7a3441d" path="/var/lib/kubelet/pods/18375e9b-7089-4b3a-b691-bd0ba7a3441d/volumes"
Oct 29 23:14:06.173117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:06.173082    6203 generic.go:356] "Generic (PLEG): container finished" podID="f8f43368-e2d8-4fca-8f89-e3319058b079" containerID="f78ffe41c0025aa0b103a98a243a31575017b541a6f4ed97553794cd4c5969ae" exitCode=0
Oct 29 23:14:06.173579 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:06.173134    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerDied","Data":"f78ffe41c0025aa0b103a98a243a31575017b541a6f4ed97553794cd4c5969ae"}
Oct 29 23:14:06.173579 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:06.173183    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerStarted","Data":"720067f2027159b7b2c3111957de0e72008be5f5a67c6cec85dfcf9b8bf36a96"}
Oct 29 23:14:07.179954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:07.179915    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerStarted","Data":"43d23e202456d88dc5c4fc067600ddbb4132a554eb2a6575f2c5bb2b7008161e"}
Oct 29 23:14:07.180419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:07.180008    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerStarted","Data":"25ab3a1df70b8660ea896ebeeb32613a40e75d3bc3de148e82c629106875c6df"}
Oct 29 23:14:07.180419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:07.180035    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerStarted","Data":"20ee386d4fffdfb336ae4d0c586d4149a3ba0170b9172af64e16f8eca45f8e50"}
Oct 29 23:14:07.180419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:07.180062    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerStarted","Data":"32dc46dd85f1812a5dfbad78dca018fc168b22c105fe4e7939a14a0ef142286b"}
Oct 29 23:14:07.180419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:07.180089    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerStarted","Data":"8aacb43230180f08faa5eac0056344e6aecd1d55bb1042cd0e54af7378659b9e"}
Oct 29 23:14:07.180419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:07.180112    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8f43368-e2d8-4fca-8f89-e3319058b079","Type":"ContainerStarted","Data":"c9b24aa38b73e2f9e82f790260f9fe5308b926941debeb0ce3dce11987f78548"}
Oct 29 23:14:07.254528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:07.254477    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.234187641 podStartE2EDuration="2.254464086s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.276445ms" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:07.253262654 +0000 UTC m=+25.357523040" watchObservedRunningTime="2026-10-29 23:14:07.254464086 +0000 UTC m=+25.358724462"
Oct 29 23:14:09.922222 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:14:09.922183    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:14:10.599759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:10.599717    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:14.383719 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:14.383688    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b97hf"
Oct 29 23:14:39.922894 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:14:39.922843    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:14:49.735675 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:49.735615    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:49.751239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:49.751204    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:14:50.340836 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:14:50.340811    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 29 23:15:09.922374 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:15:09.922336    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:15:39.922493 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:15:39.922408    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:16:09.922176 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:16:09.922135    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:16:39.922124 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:16:39.922084    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:17:09.922426 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:17:09.922349    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:17:39.922027 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:17:39.921988    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:18:09.922402 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:18:09.922363    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:18:39.922047 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:18:39.921961    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:19:09.922335 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:19:09.922298    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:19:39.921840 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:19:39.921801    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:19:44.308475 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.308439    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-2f2w5"]
Oct 29 23:19:44.310684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.310667    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.342445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.342413    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 23:19:44.342445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.342415    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-r7vwq\""
Oct 29 23:19:44.342631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.342458    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 23:19:44.342862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.342844    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2f2w5"]
Oct 29 23:19:44.430251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.430215    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v22s7\" (UniqueName: \"kubernetes.io/projected/8a2c1529-baf3-44ee-8b8a-b501a3eadc82-kube-api-access-v22s7\") pod \"cert-manager-597b95cb84-2f2w5\" (UID: \"8a2c1529-baf3-44ee-8b8a-b501a3eadc82\") " pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.430251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.430252    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a2c1529-baf3-44ee-8b8a-b501a3eadc82-bound-sa-token\") pod \"cert-manager-597b95cb84-2f2w5\" (UID: \"8a2c1529-baf3-44ee-8b8a-b501a3eadc82\") " pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.531207 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.531169    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v22s7\" (UniqueName: \"kubernetes.io/projected/8a2c1529-baf3-44ee-8b8a-b501a3eadc82-kube-api-access-v22s7\") pod \"cert-manager-597b95cb84-2f2w5\" (UID: \"8a2c1529-baf3-44ee-8b8a-b501a3eadc82\") " pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.531379 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.531211    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a2c1529-baf3-44ee-8b8a-b501a3eadc82-bound-sa-token\") pod \"cert-manager-597b95cb84-2f2w5\" (UID: \"8a2c1529-baf3-44ee-8b8a-b501a3eadc82\") " pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.556872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.556835    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a2c1529-baf3-44ee-8b8a-b501a3eadc82-bound-sa-token\") pod \"cert-manager-597b95cb84-2f2w5\" (UID: \"8a2c1529-baf3-44ee-8b8a-b501a3eadc82\") " pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.557023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.556972    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v22s7\" (UniqueName: \"kubernetes.io/projected/8a2c1529-baf3-44ee-8b8a-b501a3eadc82-kube-api-access-v22s7\") pod \"cert-manager-597b95cb84-2f2w5\" (UID: \"8a2c1529-baf3-44ee-8b8a-b501a3eadc82\") " pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.635940 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.635836    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-2f2w5"
Oct 29 23:19:44.930031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.930011    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-2f2w5"]
Oct 29 23:19:44.933086 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:19:44.933058    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a2c1529_baf3_44ee_8b8a_b501a3eadc82.slice/crio-ea52f356f52e0e8faaaefa197681f96eb186d237ba4e64b2bd428c4a69101708 WatchSource:0}: Error finding container ea52f356f52e0e8faaaefa197681f96eb186d237ba4e64b2bd428c4a69101708: Status 404 returned error can't find the container with id ea52f356f52e0e8faaaefa197681f96eb186d237ba4e64b2bd428c4a69101708
Oct 29 23:19:44.935113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:44.935093    6203 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:19:45.303289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:45.303212    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2f2w5" event={"ID":"8a2c1529-baf3-44ee-8b8a-b501a3eadc82","Type":"ContainerStarted","Data":"ea52f356f52e0e8faaaefa197681f96eb186d237ba4e64b2bd428c4a69101708"}
Oct 29 23:19:48.316229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:48.316185    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-2f2w5" event={"ID":"8a2c1529-baf3-44ee-8b8a-b501a3eadc82","Type":"ContainerStarted","Data":"8204abb97fc66f36b48e4de04f7ab2634c1ddae623bd6c291b9b8295bf909c2f"}
Oct 29 23:19:48.361267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:48.361210    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-2f2w5" podStartSLOduration=1.492760163 podStartE2EDuration="4.361196958s" totalImagesPullingTime="2.868436795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:48.36068936 +0000 UTC m=+366.464949745" watchObservedRunningTime="2026-10-29 23:19:48.361196958 +0000 UTC m=+366.465457343"
Oct 29 23:19:51.918384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.918347    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"]
Oct 29 23:19:51.920826 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.920801    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:51.949001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.948973    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 29 23:19:51.963509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.963484    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 29 23:19:51.963509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.963503    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 29 23:19:51.963711 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.963596    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9j7fg\""
Oct 29 23:19:51.963711 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.963651    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 29 23:19:51.963711 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.963665    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 29 23:19:51.964259 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.964243    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"]
Oct 29 23:19:51.981196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.981175    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hplgq\" (UniqueName: \"kubernetes.io/projected/f36b0b78-ecb5-43b7-aba7-3ff139399e60-kube-api-access-hplgq\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:51.981298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.981221    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f36b0b78-ecb5-43b7-aba7-3ff139399e60-metrics-cert\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:51.981347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.981303    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f36b0b78-ecb5-43b7-aba7-3ff139399e60-manager-config\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:51.981382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:51.981355    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f36b0b78-ecb5-43b7-aba7-3ff139399e60-cert\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.081675 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.081636    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f36b0b78-ecb5-43b7-aba7-3ff139399e60-metrics-cert\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.081860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.081688    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f36b0b78-ecb5-43b7-aba7-3ff139399e60-manager-config\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.081860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.081718    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f36b0b78-ecb5-43b7-aba7-3ff139399e60-cert\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.081860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.081746    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplgq\" (UniqueName: \"kubernetes.io/projected/f36b0b78-ecb5-43b7-aba7-3ff139399e60-kube-api-access-hplgq\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.082464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.082431    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f36b0b78-ecb5-43b7-aba7-3ff139399e60-manager-config\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.084333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.084306    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f36b0b78-ecb5-43b7-aba7-3ff139399e60-cert\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.084450 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.084426    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f36b0b78-ecb5-43b7-aba7-3ff139399e60-metrics-cert\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.107631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.107600    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplgq\" (UniqueName: \"kubernetes.io/projected/f36b0b78-ecb5-43b7-aba7-3ff139399e60-kube-api-access-hplgq\") pod \"lws-controller-manager-848684cc69-jqffg\" (UID: \"f36b0b78-ecb5-43b7-aba7-3ff139399e60\") " pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.231309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.231227    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:52.523811 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:52.523750    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"]
Oct 29 23:19:52.525848 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:19:52.525819    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36b0b78_ecb5_43b7_aba7_3ff139399e60.slice/crio-159b6e3c36e2682edbf25a9d9077168efed11508fe574c54448d597dd6d0df7c WatchSource:0}: Error finding container 159b6e3c36e2682edbf25a9d9077168efed11508fe574c54448d597dd6d0df7c: Status 404 returned error can't find the container with id 159b6e3c36e2682edbf25a9d9077168efed11508fe574c54448d597dd6d0df7c
Oct 29 23:19:53.338313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:53.338267    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg" event={"ID":"f36b0b78-ecb5-43b7-aba7-3ff139399e60","Type":"ContainerStarted","Data":"159b6e3c36e2682edbf25a9d9077168efed11508fe574c54448d597dd6d0df7c"}
Oct 29 23:19:55.346546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:55.346454    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg" event={"ID":"f36b0b78-ecb5-43b7-aba7-3ff139399e60","Type":"ContainerStarted","Data":"3388e32b276c0166d67cf1879ed673fa37ea5df16e0046c2dc231574eaaac641"}
Oct 29 23:19:55.346947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:55.346548    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:19:55.398588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:19:55.398523    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg" podStartSLOduration=1.9976221870000002 podStartE2EDuration="4.398508355s" totalImagesPullingTime="2.400886168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:19:55.398465878 +0000 UTC m=+373.502726265" watchObservedRunningTime="2026-10-29 23:19:55.398508355 +0000 UTC m=+373.502768806"
Oct 29 23:20:01.070966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.070868    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"]
Oct 29 23:20:01.074374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.074356    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.130171 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.130138    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"]
Oct 29 23:20:01.153603 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153567    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-tmp\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.153772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153618    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-manifests\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.153772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153698    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.153855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153787    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-metrics-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.153855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153810    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 23:20:01.153855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153825    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5xz\" (UniqueName: \"kubernetes.io/projected/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-kube-api-access-qs5xz\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.153855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153830    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 23:20:01.153855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153830    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-295kq\""
Oct 29 23:20:01.154052 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153843    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 29 23:20:01.154052 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.153931    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.160834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.160819    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 23:20:01.161156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.161141    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 23:20:01.254960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.254922    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-tmp\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.254960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.254961    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-manifests\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.255201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.254980    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.255201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.255027    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-metrics-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.255201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.255054    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5xz\" (UniqueName: \"kubernetes.io/projected/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-kube-api-access-qs5xz\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.255201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.255090    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.255500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.255457    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-manifests\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.257305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.257279    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-tmp\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.257564 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.257544    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.257655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.257631    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-metrics-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.257713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.257651    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-webhook-cert\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.281208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.281180    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5xz\" (UniqueName: \"kubernetes.io/projected/36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c-kube-api-access-qs5xz\") pod \"opendatahub-operator-controller-manager-57666cc899-7j9mq\" (UID: \"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c\") " pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.388585 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.388493    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:01.739136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:01.739111    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"]
Oct 29 23:20:01.741065 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:20:01.741041    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ad64d8_da8d_40e7_b3fc_a0d28f2b8e2c.slice/crio-4438325fecf385609a5c35aa42aebf7360249b8b437d91bc542808ab49acd8b5 WatchSource:0}: Error finding container 4438325fecf385609a5c35aa42aebf7360249b8b437d91bc542808ab49acd8b5: Status 404 returned error can't find the container with id 4438325fecf385609a5c35aa42aebf7360249b8b437d91bc542808ab49acd8b5
Oct 29 23:20:02.373818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:02.373738    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq" event={"ID":"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c","Type":"ContainerStarted","Data":"4438325fecf385609a5c35aa42aebf7360249b8b437d91bc542808ab49acd8b5"}
Oct 29 23:20:05.391840 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:05.391757    6203 generic.go:356] "Generic (PLEG): container finished" podID="36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c" containerID="c8e0cdd32a63ead255ecdfa5773627bfc393aa3c380424dc688297df771f2b04" exitCode=0
Oct 29 23:20:05.392251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:05.391832    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq" event={"ID":"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c","Type":"ContainerDied","Data":"c8e0cdd32a63ead255ecdfa5773627bfc393aa3c380424dc688297df771f2b04"}
Oct 29 23:20:06.352939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:06.352857    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-848684cc69-jqffg"
Oct 29 23:20:06.398220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:06.398181    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq" event={"ID":"36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c","Type":"ContainerStarted","Data":"6d04d7e57ede1d2767610742481dc653a2a836622a670d3850531382a93171c4"}
Oct 29 23:20:06.398949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:06.398236    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:06.523517 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:06.523463    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq" podStartSLOduration=2.889958488 podStartE2EDuration="6.52344909s" totalImagesPullingTime="3.195113242s" totalInitContainerRuntime="438.37736ms" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:06.508593528 +0000 UTC m=+384.612853913" watchObservedRunningTime="2026-10-29 23:20:06.52344909 +0000 UTC m=+384.627709476"
Oct 29 23:20:09.922586 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:20:09.922549    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:20:17.405078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:17.405052    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57666cc899-7j9mq"
Oct 29 23:20:31.799501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.799464    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"]
Oct 29 23:20:31.802394 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.802372    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:31.838656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.838619    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 23:20:31.838820 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.838707    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 23:20:31.839489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.839467    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"]
Oct 29 23:20:31.904686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.904653    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 23:20:31.904686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.904653    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jwvmc\""
Oct 29 23:20:31.904894 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.904726    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 23:20:31.920171 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.920139    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxbr\" (UniqueName: \"kubernetes.io/projected/291aac22-e1e9-4172-85ae-865139758cc6-kube-api-access-vhxbr\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:31.920301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.920262    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/291aac22-e1e9-4172-85ae-865139758cc6-tmp\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:31.920301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:31.920295    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291aac22-e1e9-4172-85ae-865139758cc6-tls-certs\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.021756 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.021725    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxbr\" (UniqueName: \"kubernetes.io/projected/291aac22-e1e9-4172-85ae-865139758cc6-kube-api-access-vhxbr\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.021974 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.021797    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/291aac22-e1e9-4172-85ae-865139758cc6-tmp\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.021974 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.021822    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291aac22-e1e9-4172-85ae-865139758cc6-tls-certs\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.024202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.024179    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/291aac22-e1e9-4172-85ae-865139758cc6-tmp\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.024319 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.024228    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291aac22-e1e9-4172-85ae-865139758cc6-tls-certs\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.056229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.056171    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxbr\" (UniqueName: \"kubernetes.io/projected/291aac22-e1e9-4172-85ae-865139758cc6-kube-api-access-vhxbr\") pod \"kube-auth-proxy-8578b44655-5b6lj\" (UID: \"291aac22-e1e9-4172-85ae-865139758cc6\") " pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.114231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.114193    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"
Oct 29 23:20:32.408402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.408374    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8578b44655-5b6lj"]
Oct 29 23:20:32.410503 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:20:32.410474    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291aac22_e1e9_4172_85ae_865139758cc6.slice/crio-d44811582d5a645f720709d291698e02ca25b996d51b2dae5bf18195b66a5428 WatchSource:0}: Error finding container d44811582d5a645f720709d291698e02ca25b996d51b2dae5bf18195b66a5428: Status 404 returned error can't find the container with id d44811582d5a645f720709d291698e02ca25b996d51b2dae5bf18195b66a5428
Oct 29 23:20:32.498356 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:32.498315    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj" event={"ID":"291aac22-e1e9-4172-85ae-865139758cc6","Type":"ContainerStarted","Data":"d44811582d5a645f720709d291698e02ca25b996d51b2dae5bf18195b66a5428"}
Oct 29 23:20:35.513343 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:35.513306    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj" event={"ID":"291aac22-e1e9-4172-85ae-865139758cc6","Type":"ContainerStarted","Data":"2071b0045af3ed3a71f0c19433d2916a04e20f5c8b5052b3f301edffc59df909"}
Oct 29 23:20:35.559949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:20:35.559864    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8578b44655-5b6lj" podStartSLOduration=2.192626677 podStartE2EDuration="4.559847107s" totalImagesPullingTime="2.36722043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:20:35.559453765 +0000 UTC m=+413.663714151" watchObservedRunningTime="2026-10-29 23:20:35.559847107 +0000 UTC m=+413.664107493"
Oct 29 23:20:39.922677 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:20:39.922637    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:21:09.922605 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:21:09.922560    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:21:24.848240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.848162    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-46ppt"]
Oct 29 23:21:24.852345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.852325    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:24.894858 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.894832    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 29 23:21:24.895035 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.895018    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 29 23:21:24.896278 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.896255    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-46ppt"]
Oct 29 23:21:24.980101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.980071    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5x5g7\""
Oct 29 23:21:24.982427 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.982407    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4fj\" (UniqueName: \"kubernetes.io/projected/cb04d561-0325-4a58-bab5-4a255f081c2e-kube-api-access-ff4fj\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:24.982482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.982450    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb04d561-0325-4a58-bab5-4a255f081c2e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:24.982530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:24.982483    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb04d561-0325-4a58-bab5-4a255f081c2e-cert\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.082996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.082957    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4fj\" (UniqueName: \"kubernetes.io/projected/cb04d561-0325-4a58-bab5-4a255f081c2e-kube-api-access-ff4fj\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.083190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.083023    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb04d561-0325-4a58-bab5-4a255f081c2e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.083190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.083067    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb04d561-0325-4a58-bab5-4a255f081c2e-cert\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.085782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.085729    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb04d561-0325-4a58-bab5-4a255f081c2e-cert\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.085967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.085931    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb04d561-0325-4a58-bab5-4a255f081c2e-metrics-certs\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.110376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.110312    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4fj\" (UniqueName: \"kubernetes.io/projected/cb04d561-0325-4a58-bab5-4a255f081c2e-kube-api-access-ff4fj\") pod \"odh-model-controller-f87fcf56c-46ppt\" (UID: \"cb04d561-0325-4a58-bab5-4a255f081c2e\") " pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.163221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.163192    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:25.457674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.455010    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-46ppt"]
Oct 29 23:21:25.458405 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:21:25.458376    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb04d561_0325_4a58_bab5_4a255f081c2e.slice/crio-12bd87688c614b4903d7fcd4059a9541fd1c1c9bbeb47b4d797cdfb85dd78323 WatchSource:0}: Error finding container 12bd87688c614b4903d7fcd4059a9541fd1c1c9bbeb47b4d797cdfb85dd78323: Status 404 returned error can't find the container with id 12bd87688c614b4903d7fcd4059a9541fd1c1c9bbeb47b4d797cdfb85dd78323
Oct 29 23:21:25.691568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:25.691532    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-46ppt" event={"ID":"cb04d561-0325-4a58-bab5-4a255f081c2e","Type":"ContainerStarted","Data":"12bd87688c614b4903d7fcd4059a9541fd1c1c9bbeb47b4d797cdfb85dd78323"}
Oct 29 23:21:28.712978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:28.712933    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-46ppt" event={"ID":"cb04d561-0325-4a58-bab5-4a255f081c2e","Type":"ContainerStarted","Data":"2ec8afb895e098219bd3d6d60736c978af646447cb1068cefbc9be306d786302"}
Oct 29 23:21:28.713376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:28.713125    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:28.758991 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:28.758939    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-46ppt" podStartSLOduration=2.31061067 podStartE2EDuration="4.758925226s" totalImagesPullingTime="2.448314556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:28.758706367 +0000 UTC m=+466.862966754" watchObservedRunningTime="2026-10-29 23:21:28.758925226 +0000 UTC m=+466.863185603"
Oct 29 23:21:39.719047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:21:39.719021    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-46ppt"
Oct 29 23:21:39.922163 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:21:39.922122    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:22:09.922486 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:22:09.922443    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:22:35.922860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:35.922828    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"]
Oct 29 23:22:35.926382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:35.926359    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:35.947203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:35.947171    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 23:22:35.967905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:35.963488    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dllbn\""
Oct 29 23:22:35.967905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:35.963541    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 23:22:35.967905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:35.964681    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"]
Oct 29 23:22:35.990191 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:35.990157    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9nlp\" (UniqueName: \"kubernetes.io/projected/9bf28688-4e8c-4ec9-a38f-57be6759b9ab-kube-api-access-g9nlp\") pod \"limitador-operator-controller-manager-856dd584c5-w4xdx\" (UID: \"9bf28688-4e8c-4ec9-a38f-57be6759b9ab\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:36.091330 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:36.091285    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9nlp\" (UniqueName: \"kubernetes.io/projected/9bf28688-4e8c-4ec9-a38f-57be6759b9ab-kube-api-access-g9nlp\") pod \"limitador-operator-controller-manager-856dd584c5-w4xdx\" (UID: \"9bf28688-4e8c-4ec9-a38f-57be6759b9ab\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:36.118982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:36.118953    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9nlp\" (UniqueName: \"kubernetes.io/projected/9bf28688-4e8c-4ec9-a38f-57be6759b9ab-kube-api-access-g9nlp\") pod \"limitador-operator-controller-manager-856dd584c5-w4xdx\" (UID: \"9bf28688-4e8c-4ec9-a38f-57be6759b9ab\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:36.238917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:36.238809    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:36.534750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:36.534720    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"]
Oct 29 23:22:36.537571 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:22:36.537541    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf28688_4e8c_4ec9_a38f_57be6759b9ab.slice/crio-4a91f5486aaeb7f343ea53974b710e0648b7244d15881af30b74c01e82d2b39e WatchSource:0}: Error finding container 4a91f5486aaeb7f343ea53974b710e0648b7244d15881af30b74c01e82d2b39e: Status 404 returned error can't find the container with id 4a91f5486aaeb7f343ea53974b710e0648b7244d15881af30b74c01e82d2b39e
Oct 29 23:22:36.964093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:36.964048    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx" event={"ID":"9bf28688-4e8c-4ec9-a38f-57be6759b9ab","Type":"ContainerStarted","Data":"4a91f5486aaeb7f343ea53974b710e0648b7244d15881af30b74c01e82d2b39e"}
Oct 29 23:22:38.974852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:38.974809    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx" event={"ID":"9bf28688-4e8c-4ec9-a38f-57be6759b9ab","Type":"ContainerStarted","Data":"7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2"}
Oct 29 23:22:38.975285 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:38.974986    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:39.021927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:39.021803    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx" podStartSLOduration=1.795738691 podStartE2EDuration="4.021790996s" totalImagesPullingTime="2.226052305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:39.021214702 +0000 UTC m=+537.125475088" watchObservedRunningTime="2026-10-29 23:22:39.021790996 +0000 UTC m=+537.126051383"
Oct 29 23:22:39.922609 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:22:39.922500    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:22:49.168498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.168462    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"]
Oct 29 23:22:49.169145 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.168676    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" containerName="manager" containerID="cri-o://7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2" gracePeriod=2
Oct 29 23:22:49.170653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.170626    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:49.182963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.182938    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"]
Oct 29 23:22:49.184985 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:22:49.184946    6203 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w4xdx\" is forbidden: User \"system:node:ip-10-0-139-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-224.ec2.internal' and this object" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:49.195369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.195347    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"]
Oct 29 23:22:49.195681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.195668    6203 cpu_manager.go:414] "containerMap: removing container" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" containerName="manager"
Oct 29 23:22:49.195725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.195682    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" containerName="manager"
Oct 29 23:22:49.195758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.195731    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" containerName="manager"
Oct 29 23:22:49.198604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.198588    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"
Oct 29 23:22:49.202909 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:22:49.202861    6203 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w4xdx\" is forbidden: User \"system:node:ip-10-0-139-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-224.ec2.internal' and this object" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:49.222420 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:22:49.222378    6203 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w4xdx\" is forbidden: User \"system:node:ip-10-0-139-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-224.ec2.internal' and this object" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:49.231374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.231346    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"]
Oct 29 23:22:49.317613 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.316300    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wld8v\" (UniqueName: \"kubernetes.io/projected/c162a52c-5579-4b27-84de-c7dfae18774b-kube-api-access-wld8v\") pod \"limitador-operator-controller-manager-856dd584c5-mj82z\" (UID: \"c162a52c-5579-4b27-84de-c7dfae18774b\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"
Oct 29 23:22:49.417483 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.417443    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wld8v\" (UniqueName: \"kubernetes.io/projected/c162a52c-5579-4b27-84de-c7dfae18774b-kube-api-access-wld8v\") pod \"limitador-operator-controller-manager-856dd584c5-mj82z\" (UID: \"c162a52c-5579-4b27-84de-c7dfae18774b\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"
Oct 29 23:22:49.451478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.451452    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wld8v\" (UniqueName: \"kubernetes.io/projected/c162a52c-5579-4b27-84de-c7dfae18774b-kube-api-access-wld8v\") pod \"limitador-operator-controller-manager-856dd584c5-mj82z\" (UID: \"c162a52c-5579-4b27-84de-c7dfae18774b\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"
Oct 29 23:22:49.453730 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.453708    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:49.469614 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:22:49.469584    6203 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-w4xdx\" is forbidden: User \"system:node:ip-10-0-139-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-224.ec2.internal' and this object" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:49.548619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.548591    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"
Oct 29 23:22:49.618283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.618254    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf28688-4e8c-4ec9-a38f-57be6759b9ab-kube-api-access-g9nlp\" (UniqueName: \"kubernetes.io/projected/9bf28688-4e8c-4ec9-a38f-57be6759b9ab-kube-api-access-g9nlp\") pod \"9bf28688-4e8c-4ec9-a38f-57be6759b9ab\" (UID: \"9bf28688-4e8c-4ec9-a38f-57be6759b9ab\") "
Oct 29 23:22:49.620430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.620402    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf28688-4e8c-4ec9-a38f-57be6759b9ab-kube-api-access-g9nlp" pod "9bf28688-4e8c-4ec9-a38f-57be6759b9ab" (UID: "9bf28688-4e8c-4ec9-a38f-57be6759b9ab"). InnerVolumeSpecName "kube-api-access-g9nlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:22:49.719707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.719632    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9nlp\" (UniqueName: \"kubernetes.io/projected/9bf28688-4e8c-4ec9-a38f-57be6759b9ab-kube-api-access-g9nlp\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:22:49.847174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:49.847149    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"]
Oct 29 23:22:49.849212 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:22:49.849176    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc162a52c_5579_4b27_84de_c7dfae18774b.slice/crio-e06e0f2d3c1d0a876a648040b2c1329f7bc9d542cf14e7af6a250906d5b9fbdf WatchSource:0}: Error finding container e06e0f2d3c1d0a876a648040b2c1329f7bc9d542cf14e7af6a250906d5b9fbdf: Status 404 returned error can't find the container with id e06e0f2d3c1d0a876a648040b2c1329f7bc9d542cf14e7af6a250906d5b9fbdf
Oct 29 23:22:50.022143 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.022063    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" path="/var/lib/kubelet/pods/9bf28688-4e8c-4ec9-a38f-57be6759b9ab/volumes"
Oct 29 23:22:50.024042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.024003    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z" event={"ID":"c162a52c-5579-4b27-84de-c7dfae18774b","Type":"ContainerStarted","Data":"8c099f47629b0f5165b563c08f64586987d0caa50cea5caf808a452e104d71c6"}
Oct 29 23:22:50.024200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.024077    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"
Oct 29 23:22:50.024200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.024102    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z" event={"ID":"c162a52c-5579-4b27-84de-c7dfae18774b","Type":"ContainerStarted","Data":"e06e0f2d3c1d0a876a648040b2c1329f7bc9d542cf14e7af6a250906d5b9fbdf"}
Oct 29 23:22:50.025361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.025334    6203 generic.go:356] "Generic (PLEG): container finished" podID="9bf28688-4e8c-4ec9-a38f-57be6759b9ab" containerID="7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2" exitCode=0
Oct 29 23:22:50.025481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.025373    6203 scope.go:122] "RemoveContainer" containerID="7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2"
Oct 29 23:22:50.025481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.025388    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-w4xdx"
Oct 29 23:22:50.033602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.033587    6203 scope.go:122] "RemoveContainer" containerID="7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2"
Oct 29 23:22:50.033912 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:22:50.033867    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2\": container with ID starting with 7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2 not found: ID does not exist" containerID="7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2"
Oct 29 23:22:50.034016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.033919    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2"} err="failed to get container status \"7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2\": rpc error: code = NotFound desc = could not find container \"7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2\": container with ID starting with 7d3df808843990ad47772a71ebbdd5a153f74c5e15ac8d49035fb4c730a325c2 not found: ID does not exist"
Oct 29 23:22:50.071034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:22:50.070983    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z" podStartSLOduration=1.070970514 podStartE2EDuration="1.070970514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:22:50.069846453 +0000 UTC m=+548.174106839" watchObservedRunningTime="2026-10-29 23:22:50.070970514 +0000 UTC m=+548.175230897"
Oct 29 23:23:01.033062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:01.032975    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mj82z"
Oct 29 23:23:09.922592 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:23:09.922555    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:23:35.423834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.423803    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jstvl"]
Oct 29 23:23:35.427048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.427024    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jstvl"
Oct 29 23:23:35.445437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.445415    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jstvl"]
Oct 29 23:23:35.459348 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.459324    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-znb87\""
Oct 29 23:23:35.509767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.509736    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvdf\" (UniqueName: \"kubernetes.io/projected/5cee702a-b591-47fa-bc99-8d50b1e140b3-kube-api-access-wsvdf\") pod \"authorino-57f7f8659f-jstvl\" (UID: \"5cee702a-b591-47fa-bc99-8d50b1e140b3\") " pod="kuadrant-system/authorino-57f7f8659f-jstvl"
Oct 29 23:23:35.610797 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.610756    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvdf\" (UniqueName: \"kubernetes.io/projected/5cee702a-b591-47fa-bc99-8d50b1e140b3-kube-api-access-wsvdf\") pod \"authorino-57f7f8659f-jstvl\" (UID: \"5cee702a-b591-47fa-bc99-8d50b1e140b3\") " pod="kuadrant-system/authorino-57f7f8659f-jstvl"
Oct 29 23:23:35.636821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.636789    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvdf\" (UniqueName: \"kubernetes.io/projected/5cee702a-b591-47fa-bc99-8d50b1e140b3-kube-api-access-wsvdf\") pod \"authorino-57f7f8659f-jstvl\" (UID: \"5cee702a-b591-47fa-bc99-8d50b1e140b3\") " pod="kuadrant-system/authorino-57f7f8659f-jstvl"
Oct 29 23:23:35.738118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:35.738038    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jstvl"
Oct 29 23:23:36.028695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.028668    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jstvl"]
Oct 29 23:23:36.031871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:23:36.031835    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cee702a_b591_47fa_bc99_8d50b1e140b3.slice/crio-096ca7ef559d5ea8a7e8e2ca873c679e80daea8990bc18d7a576c30e36fd1076 WatchSource:0}: Error finding container 096ca7ef559d5ea8a7e8e2ca873c679e80daea8990bc18d7a576c30e36fd1076: Status 404 returned error can't find the container with id 096ca7ef559d5ea8a7e8e2ca873c679e80daea8990bc18d7a576c30e36fd1076
Oct 29 23:23:36.093386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.093348    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhkh4"]
Oct 29 23:23:36.097810 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.097785    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4"
Oct 29 23:23:36.118338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.118312    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhkh4"]
Oct 29 23:23:36.206190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.206154    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jstvl" event={"ID":"5cee702a-b591-47fa-bc99-8d50b1e140b3","Type":"ContainerStarted","Data":"096ca7ef559d5ea8a7e8e2ca873c679e80daea8990bc18d7a576c30e36fd1076"}
Oct 29 23:23:36.214620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.214592    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjg8\" (UniqueName: \"kubernetes.io/projected/b9332257-3b26-4dfe-9e90-21dafc34e137-kube-api-access-ndjg8\") pod \"authorino-56c6c9ddd5-mhkh4\" (UID: \"b9332257-3b26-4dfe-9e90-21dafc34e137\") " pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4"
Oct 29 23:23:36.315227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.315148    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjg8\" (UniqueName: \"kubernetes.io/projected/b9332257-3b26-4dfe-9e90-21dafc34e137-kube-api-access-ndjg8\") pod \"authorino-56c6c9ddd5-mhkh4\" (UID: \"b9332257-3b26-4dfe-9e90-21dafc34e137\") " pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4"
Oct 29 23:23:36.340527 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.340494    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjg8\" (UniqueName: \"kubernetes.io/projected/b9332257-3b26-4dfe-9e90-21dafc34e137-kube-api-access-ndjg8\") pod \"authorino-56c6c9ddd5-mhkh4\" (UID: \"b9332257-3b26-4dfe-9e90-21dafc34e137\") " pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4"
Oct 29 23:23:36.409928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.409894    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4"
Oct 29 23:23:36.709832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:36.709801    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhkh4"]
Oct 29 23:23:36.713780 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:23:36.713740    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9332257_3b26_4dfe_9e90_21dafc34e137.slice/crio-dc04daf4e8d9494541eb6de5b7e5545ff504a9db6f6d4a11c24bae008bf4f44b WatchSource:0}: Error finding container dc04daf4e8d9494541eb6de5b7e5545ff504a9db6f6d4a11c24bae008bf4f44b: Status 404 returned error can't find the container with id dc04daf4e8d9494541eb6de5b7e5545ff504a9db6f6d4a11c24bae008bf4f44b
Oct 29 23:23:37.213656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:37.213615    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4" event={"ID":"b9332257-3b26-4dfe-9e90-21dafc34e137","Type":"ContainerStarted","Data":"dc04daf4e8d9494541eb6de5b7e5545ff504a9db6f6d4a11c24bae008bf4f44b"}
Oct 29 23:23:38.218931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:38.218895    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4" event={"ID":"b9332257-3b26-4dfe-9e90-21dafc34e137","Type":"ContainerStarted","Data":"ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c"}
Oct 29 23:23:38.220366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:38.220338    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jstvl" event={"ID":"5cee702a-b591-47fa-bc99-8d50b1e140b3","Type":"ContainerStarted","Data":"4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6"}
Oct 29 23:23:38.263231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:38.263186    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4" podStartSLOduration=1.305052957 podStartE2EDuration="2.263173926s" totalImagesPullingTime="958.120969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:38.26203109 +0000 UTC m=+596.366291476" watchObservedRunningTime="2026-10-29 23:23:38.263173926 +0000 UTC m=+596.367434313"
Oct 29 23:23:38.287653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:38.287621    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jstvl"]
Oct 29 23:23:38.308824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:38.308776    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-jstvl" podStartSLOduration=1.714545784 podStartE2EDuration="3.308763211s" totalImagesPullingTime="1.594217427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:23:38.308072382 +0000 UTC m=+596.412332769" watchObservedRunningTime="2026-10-29 23:23:38.308763211 +0000 UTC m=+596.413023596"
Oct 29 23:23:39.923100 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:23:39.923061    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:23:40.229518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:40.229409    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-jstvl" podUID="5cee702a-b591-47fa-bc99-8d50b1e140b3" containerName="authorino" containerID="cri-o://4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6" gracePeriod=30
Oct 29 23:23:40.464055 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:40.464021    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jstvl"
Oct 29 23:23:40.552136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:40.552048    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cee702a-b591-47fa-bc99-8d50b1e140b3-kube-api-access-wsvdf\" (UniqueName: \"kubernetes.io/projected/5cee702a-b591-47fa-bc99-8d50b1e140b3-kube-api-access-wsvdf\") pod \"5cee702a-b591-47fa-bc99-8d50b1e140b3\" (UID: \"5cee702a-b591-47fa-bc99-8d50b1e140b3\") "
Oct 29 23:23:40.554099 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:40.554073    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cee702a-b591-47fa-bc99-8d50b1e140b3-kube-api-access-wsvdf" pod "5cee702a-b591-47fa-bc99-8d50b1e140b3" (UID: "5cee702a-b591-47fa-bc99-8d50b1e140b3"). InnerVolumeSpecName "kube-api-access-wsvdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:23:40.653327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:40.653286    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvdf\" (UniqueName: \"kubernetes.io/projected/5cee702a-b591-47fa-bc99-8d50b1e140b3-kube-api-access-wsvdf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:23:41.235456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.235417    6203 generic.go:356] "Generic (PLEG): container finished" podID="5cee702a-b591-47fa-bc99-8d50b1e140b3" containerID="4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6" exitCode=0
Oct 29 23:23:41.235872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.235467    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jstvl"
Oct 29 23:23:41.235872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.235469    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jstvl" event={"ID":"5cee702a-b591-47fa-bc99-8d50b1e140b3","Type":"ContainerDied","Data":"4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6"}
Oct 29 23:23:41.235872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.235585    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jstvl" event={"ID":"5cee702a-b591-47fa-bc99-8d50b1e140b3","Type":"ContainerDied","Data":"096ca7ef559d5ea8a7e8e2ca873c679e80daea8990bc18d7a576c30e36fd1076"}
Oct 29 23:23:41.235872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.235605    6203 scope.go:122] "RemoveContainer" containerID="4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6"
Oct 29 23:23:41.244238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.244211    6203 scope.go:122] "RemoveContainer" containerID="4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6"
Oct 29 23:23:41.244506 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:23:41.244486    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6\": container with ID starting with 4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6 not found: ID does not exist" containerID="4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6"
Oct 29 23:23:41.244577 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.244520    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6"} err="failed to get container status \"4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6\": rpc error: code = NotFound desc = could not find container \"4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6\": container with ID starting with 4087c9637ea48fbc472bec8b82f6d52b34b11664578ed8490bbb8fb79d3483a6 not found: ID does not exist"
Oct 29 23:23:41.294366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.294327    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jstvl"]
Oct 29 23:23:41.298995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:41.298968    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jstvl"]
Oct 29 23:23:42.016279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:23:42.016245    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cee702a-b591-47fa-bc99-8d50b1e140b3" path="/var/lib/kubelet/pods/5cee702a-b591-47fa-bc99-8d50b1e140b3/volumes"
Oct 29 23:24:08.528518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.528483    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nfbd9"]
Oct 29 23:24:08.529021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.528807    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5cee702a-b591-47fa-bc99-8d50b1e140b3" containerName="authorino"
Oct 29 23:24:08.529021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.528817    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cee702a-b591-47fa-bc99-8d50b1e140b3" containerName="authorino"
Oct 29 23:24:08.529021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.528864    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cee702a-b591-47fa-bc99-8d50b1e140b3" containerName="authorino"
Oct 29 23:24:08.531841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.531826    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nfbd9"
Oct 29 23:24:08.550650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.550622    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nfbd9"]
Oct 29 23:24:08.605778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.605748    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5rs\" (UniqueName: \"kubernetes.io/projected/3cfa9433-1aaa-429c-bb9b-8968e84ef627-kube-api-access-cz5rs\") pod \"authorino-6f6944bcdd-nfbd9\" (UID: \"3cfa9433-1aaa-429c-bb9b-8968e84ef627\") " pod="kuadrant-system/authorino-6f6944bcdd-nfbd9"
Oct 29 23:24:08.707165 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.707126    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5rs\" (UniqueName: \"kubernetes.io/projected/3cfa9433-1aaa-429c-bb9b-8968e84ef627-kube-api-access-cz5rs\") pod \"authorino-6f6944bcdd-nfbd9\" (UID: \"3cfa9433-1aaa-429c-bb9b-8968e84ef627\") " pod="kuadrant-system/authorino-6f6944bcdd-nfbd9"
Oct 29 23:24:08.724998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.724967    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nfbd9"]
Oct 29 23:24:08.725188 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:24:08.725169    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cz5rs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-nfbd9" podUID="3cfa9433-1aaa-429c-bb9b-8968e84ef627"
Oct 29 23:24:08.732556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.732526    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5rs\" (UniqueName: \"kubernetes.io/projected/3cfa9433-1aaa-429c-bb9b-8968e84ef627-kube-api-access-cz5rs\") pod \"authorino-6f6944bcdd-nfbd9\" (UID: \"3cfa9433-1aaa-429c-bb9b-8968e84ef627\") " pod="kuadrant-system/authorino-6f6944bcdd-nfbd9"
Oct 29 23:24:08.750296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.750272    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgdt9"]
Oct 29 23:24:08.753525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.753508    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:08.772054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.772019    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 29 23:24:08.772750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.772725    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgdt9"]
Oct 29 23:24:08.808264 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.808195    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-tls-cert\") pod \"authorino-6889cbd8c4-qgdt9\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") " pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:08.808264 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.808234    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84knl\" (UniqueName: \"kubernetes.io/projected/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-kube-api-access-84knl\") pod \"authorino-6889cbd8c4-qgdt9\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") " pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:08.850655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.850625    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgdt9"]
Oct 29 23:24:08.850844 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:24:08.850825    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-84knl tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-qgdt9" podUID="e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b"
Oct 29 23:24:08.877499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.877468    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-576ff54d58-ms2ph"]
Oct 29 23:24:08.880738 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.880720    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:08.901918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.901893    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-576ff54d58-ms2ph"]
Oct 29 23:24:08.909535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.909512    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-tls-cert\") pod \"authorino-6889cbd8c4-qgdt9\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") " pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:08.909621 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.909545    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84knl\" (UniqueName: \"kubernetes.io/projected/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-kube-api-access-84knl\") pod \"authorino-6889cbd8c4-qgdt9\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") " pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:08.911953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.911930    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-tls-cert\") pod \"authorino-6889cbd8c4-qgdt9\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") " pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:08.935257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:08.935228    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84knl\" (UniqueName: \"kubernetes.io/projected/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-kube-api-access-84knl\") pod \"authorino-6889cbd8c4-qgdt9\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") " pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:09.010527 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.010496    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b72e167c-2336-4b08-aeb8-ba505de13466-tls-cert\") pod \"authorino-576ff54d58-ms2ph\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") " pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:09.010655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.010551    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngq8s\" (UniqueName: \"kubernetes.io/projected/b72e167c-2336-4b08-aeb8-ba505de13466-kube-api-access-ngq8s\") pod \"authorino-576ff54d58-ms2ph\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") " pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:09.111931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.111837    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngq8s\" (UniqueName: \"kubernetes.io/projected/b72e167c-2336-4b08-aeb8-ba505de13466-kube-api-access-ngq8s\") pod \"authorino-576ff54d58-ms2ph\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") " pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:09.112061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.111937    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b72e167c-2336-4b08-aeb8-ba505de13466-tls-cert\") pod \"authorino-576ff54d58-ms2ph\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") " pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:09.114356 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.114328    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b72e167c-2336-4b08-aeb8-ba505de13466-tls-cert\") pod \"authorino-576ff54d58-ms2ph\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") " pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:09.137279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.137251    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngq8s\" (UniqueName: \"kubernetes.io/projected/b72e167c-2336-4b08-aeb8-ba505de13466-kube-api-access-ngq8s\") pod \"authorino-576ff54d58-ms2ph\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") " pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:09.192175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.192140    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:24:09.351366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.351334    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:09.351545 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.351337    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nfbd9"
Oct 29 23:24:09.358750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.358726    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:09.361565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.361546    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nfbd9"
Oct 29 23:24:09.414636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.414580    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-kube-api-access-84knl\" (UniqueName: \"kubernetes.io/projected/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-kube-api-access-84knl\") pod \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") "
Oct 29 23:24:09.414636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.414608    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfa9433-1aaa-429c-bb9b-8968e84ef627-kube-api-access-cz5rs\" (UniqueName: \"kubernetes.io/projected/3cfa9433-1aaa-429c-bb9b-8968e84ef627-kube-api-access-cz5rs\") pod \"3cfa9433-1aaa-429c-bb9b-8968e84ef627\" (UID: \"3cfa9433-1aaa-429c-bb9b-8968e84ef627\") "
Oct 29 23:24:09.414778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.414653    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-tls-cert\" (UniqueName: \"kubernetes.io/secret/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-tls-cert\") pod \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\" (UID: \"e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b\") "
Oct 29 23:24:09.416662 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.416632    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfa9433-1aaa-429c-bb9b-8968e84ef627-kube-api-access-cz5rs" pod "3cfa9433-1aaa-429c-bb9b-8968e84ef627" (UID: "3cfa9433-1aaa-429c-bb9b-8968e84ef627"). InnerVolumeSpecName "kube-api-access-cz5rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:09.416765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.416728    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-kube-api-access-84knl" pod "e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b" (UID: "e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b"). InnerVolumeSpecName "kube-api-access-84knl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:09.416765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.416737    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-tls-cert" pod "e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b" (UID: "e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:24:09.484639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.484616    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-576ff54d58-ms2ph"]
Oct 29 23:24:09.487073 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:24:09.487040    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb72e167c_2336_4b08_aeb8_ba505de13466.slice/crio-259b4eb23e6cdcc631ec3a6723f6e66f22e25196a91a6275846ba738610fa214 WatchSource:0}: Error finding container 259b4eb23e6cdcc631ec3a6723f6e66f22e25196a91a6275846ba738610fa214: Status 404 returned error can't find the container with id 259b4eb23e6cdcc631ec3a6723f6e66f22e25196a91a6275846ba738610fa214
Oct 29 23:24:09.516033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.516010    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84knl\" (UniqueName: \"kubernetes.io/projected/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-kube-api-access-84knl\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:24:09.516033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.516031    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5rs\" (UniqueName: \"kubernetes.io/projected/3cfa9433-1aaa-429c-bb9b-8968e84ef627-kube-api-access-cz5rs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:24:09.516189 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:09.516042    6203 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b-tls-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:24:09.923057 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:24:09.923025    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:24:10.357264 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.357170    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576ff54d58-ms2ph" event={"ID":"b72e167c-2336-4b08-aeb8-ba505de13466","Type":"ContainerStarted","Data":"783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4"}
Oct 29 23:24:10.357264 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.357222    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576ff54d58-ms2ph" event={"ID":"b72e167c-2336-4b08-aeb8-ba505de13466","Type":"ContainerStarted","Data":"259b4eb23e6cdcc631ec3a6723f6e66f22e25196a91a6275846ba738610fa214"}
Oct 29 23:24:10.357469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.357283    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-qgdt9"
Oct 29 23:24:10.357469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.357374    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nfbd9"
Oct 29 23:24:10.401736 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.401678    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-576ff54d58-ms2ph" podStartSLOduration=1.855360611 podStartE2EDuration="2.401664447s" totalImagesPullingTime="546.303836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:24:10.40133656 +0000 UTC m=+628.505596946" watchObservedRunningTime="2026-10-29 23:24:10.401664447 +0000 UTC m=+628.505924832"
Oct 29 23:24:10.429312 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.429280    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhkh4"]
Oct 29 23:24:10.429512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.429488    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4" podUID="b9332257-3b26-4dfe-9e90-21dafc34e137" containerName="authorino" containerID="cri-o://ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c" gracePeriod=30
Oct 29 23:24:10.470223 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.470192    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nfbd9"]
Oct 29 23:24:10.475068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.475034    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nfbd9"]
Oct 29 23:24:10.539862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.539829    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgdt9"]
Oct 29 23:24:10.541549 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.541524    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-qgdt9"]
Oct 29 23:24:10.657251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.657224    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4"
Oct 29 23:24:10.726593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.726560    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9332257-3b26-4dfe-9e90-21dafc34e137-kube-api-access-ndjg8\" (UniqueName: \"kubernetes.io/projected/b9332257-3b26-4dfe-9e90-21dafc34e137-kube-api-access-ndjg8\") pod \"b9332257-3b26-4dfe-9e90-21dafc34e137\" (UID: \"b9332257-3b26-4dfe-9e90-21dafc34e137\") "
Oct 29 23:24:10.728601 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.728578    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9332257-3b26-4dfe-9e90-21dafc34e137-kube-api-access-ndjg8" pod "b9332257-3b26-4dfe-9e90-21dafc34e137" (UID: "b9332257-3b26-4dfe-9e90-21dafc34e137"). InnerVolumeSpecName "kube-api-access-ndjg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:24:10.827141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:10.827096    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjg8\" (UniqueName: \"kubernetes.io/projected/b9332257-3b26-4dfe-9e90-21dafc34e137-kube-api-access-ndjg8\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:24:11.362804 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.362766    6203 generic.go:356] "Generic (PLEG): container finished" podID="b9332257-3b26-4dfe-9e90-21dafc34e137" containerID="ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c" exitCode=0
Oct 29 23:24:11.363298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.362813    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4"
Oct 29 23:24:11.363298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.362853    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4" event={"ID":"b9332257-3b26-4dfe-9e90-21dafc34e137","Type":"ContainerDied","Data":"ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c"}
Oct 29 23:24:11.363298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.362916    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mhkh4" event={"ID":"b9332257-3b26-4dfe-9e90-21dafc34e137","Type":"ContainerDied","Data":"dc04daf4e8d9494541eb6de5b7e5545ff504a9db6f6d4a11c24bae008bf4f44b"}
Oct 29 23:24:11.363298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.362934    6203 scope.go:122] "RemoveContainer" containerID="ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c"
Oct 29 23:24:11.373989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.373966    6203 scope.go:122] "RemoveContainer" containerID="ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c"
Oct 29 23:24:11.374485 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:24:11.374460    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c\": container with ID starting with ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c not found: ID does not exist" containerID="ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c"
Oct 29 23:24:11.374598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.374491    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c"} err="failed to get container status \"ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c\": rpc error: code = NotFound desc = could not find container \"ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c\": container with ID starting with ab117c511877518e802af9ad414c4d57412cb1fdb3b4f69854d2b576567ba56c not found: ID does not exist"
Oct 29 23:24:11.425397 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.425355    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhkh4"]
Oct 29 23:24:11.429800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:11.429775    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mhkh4"]
Oct 29 23:24:12.018641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:12.018606    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfa9433-1aaa-429c-bb9b-8968e84ef627" path="/var/lib/kubelet/pods/3cfa9433-1aaa-429c-bb9b-8968e84ef627/volumes"
Oct 29 23:24:12.018836 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:12.018824    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9332257-3b26-4dfe-9e90-21dafc34e137" path="/var/lib/kubelet/pods/b9332257-3b26-4dfe-9e90-21dafc34e137/volumes"
Oct 29 23:24:12.019173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:24:12.019161    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b" path="/var/lib/kubelet/pods/e6f9442f-9e5d-4cf4-aeef-8cd63ca09d2b/volumes"
Oct 29 23:24:39.923062 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:24:39.922980    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:25:09.921530 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:25:09.921490    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:25:39.922333 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:25:39.922293    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:25:57.863298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.863085    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"]
Oct 29 23:25:57.863850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.863598    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b9332257-3b26-4dfe-9e90-21dafc34e137" containerName="authorino"
Oct 29 23:25:57.863850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.863617    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9332257-3b26-4dfe-9e90-21dafc34e137" containerName="authorino"
Oct 29 23:25:57.863850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.863733    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9332257-3b26-4dfe-9e90-21dafc34e137" containerName="authorino"
Oct 29 23:25:57.866768 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.866746    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.888422 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.888394    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 29 23:25:57.896320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.896297    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 23:25:57.897940 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.897920    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.898009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.897947    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.898009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.897971    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.898122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.898098    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4fv\" (UniqueName: \"kubernetes.io/projected/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-kube-api-access-vs4fv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.898170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.898136    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.898205 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.898191    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.902914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.902868    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 23:25:57.903782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.903766    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"]
Oct 29 23:25:57.988031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.988001    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k87bg\""
Oct 29 23:25:57.999446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.999415    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.999576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.999449    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.999576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.999471    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.999576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.999503    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4fv\" (UniqueName: \"kubernetes.io/projected/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-kube-api-access-vs4fv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.999576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.999531    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.999576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.999566    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:57.999991 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:57.999968    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:58.000097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.000028    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:58.000097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.000049    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:58.001804 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.001783    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:58.001942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.001925    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:58.026549 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.026514    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4fv\" (UniqueName: \"kubernetes.io/projected/415d9077-00ae-4e22-8eed-8e1e0f94e1b1-kube-api-access-vs4fv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6\" (UID: \"415d9077-00ae-4e22-8eed-8e1e0f94e1b1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:58.180018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.179988    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:25:58.478148 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.478123    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"]
Oct 29 23:25:58.480676 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:25:58.480638    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415d9077_00ae_4e22_8eed_8e1e0f94e1b1.slice/crio-fb5b9197636ae509997050b902737f1603263fdf0e94335f267ae4d6800960f2 WatchSource:0}: Error finding container fb5b9197636ae509997050b902737f1603263fdf0e94335f267ae4d6800960f2: Status 404 returned error can't find the container with id fb5b9197636ae509997050b902737f1603263fdf0e94335f267ae4d6800960f2
Oct 29 23:25:58.481847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.481833    6203 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:25:58.801098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:58.801002    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6" event={"ID":"415d9077-00ae-4e22-8eed-8e1e0f94e1b1","Type":"ContainerStarted","Data":"fb5b9197636ae509997050b902737f1603263fdf0e94335f267ae4d6800960f2"}
Oct 29 23:25:59.782513 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.782479    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"]
Oct 29 23:25:59.787581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.787558    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.807733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.807707    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 29 23:25:59.816130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.816103    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.816246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.816173    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdtjf\" (UniqueName: \"kubernetes.io/projected/8f451549-e817-46f9-bf9c-50f78d4c9091-kube-api-access-bdtjf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.816246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.816198    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.816246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.816225    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.816410 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.816285    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f451549-e817-46f9-bf9c-50f78d4c9091-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.816410 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.816325    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.822979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.822948    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"]
Oct 29 23:25:59.917541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.917501    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.917541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.917541    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.917797 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.917587    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f451549-e817-46f9-bf9c-50f78d4c9091-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.917797 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.917613    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.917797 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.917648    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.917797 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.917707    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdtjf\" (UniqueName: \"kubernetes.io/projected/8f451549-e817-46f9-bf9c-50f78d4c9091-kube-api-access-bdtjf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.918160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.918104    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.918297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.918249    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.918297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.918262    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.919869 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.919849    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f451549-e817-46f9-bf9c-50f78d4c9091-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.920229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.920210    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f451549-e817-46f9-bf9c-50f78d4c9091-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:25:59.950821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:25:59.950793    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdtjf\" (UniqueName: \"kubernetes.io/projected/8f451549-e817-46f9-bf9c-50f78d4c9091-kube-api-access-bdtjf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2\" (UID: \"8f451549-e817-46f9-bf9c-50f78d4c9091\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:26:00.110109 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:00.110024    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:26:00.407031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:00.407009    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"]
Oct 29 23:26:00.409535 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:00.409495    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f451549_e817_46f9_bf9c_50f78d4c9091.slice/crio-4d2ef32d10564cfa71d0ab0740b962992017c01d482a0fbabd6c504fb287e821 WatchSource:0}: Error finding container 4d2ef32d10564cfa71d0ab0740b962992017c01d482a0fbabd6c504fb287e821: Status 404 returned error can't find the container with id 4d2ef32d10564cfa71d0ab0740b962992017c01d482a0fbabd6c504fb287e821
Oct 29 23:26:00.812358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:00.812318    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6" event={"ID":"415d9077-00ae-4e22-8eed-8e1e0f94e1b1","Type":"ContainerStarted","Data":"29883af606b41693cc3a507ec956255904c0a082107e4fd20216f898c8affbef"}
Oct 29 23:26:00.813908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:00.813864    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2" event={"ID":"8f451549-e817-46f9-bf9c-50f78d4c9091","Type":"ContainerStarted","Data":"f112db5feec9a7d95fdc0d204f2747d6b73355b3669340528da61e7ed353e787"}
Oct 29 23:26:00.814034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:00.813921    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2" event={"ID":"8f451549-e817-46f9-bf9c-50f78d4c9091","Type":"ContainerStarted","Data":"4d2ef32d10564cfa71d0ab0740b962992017c01d482a0fbabd6c504fb287e821"}
Oct 29 23:26:00.860915 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:00.860854    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6" podStartSLOduration=2.593142345 podStartE2EDuration="3.860839819s" totalImagesPullingTime="1.267697474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:00.859792934 +0000 UTC m=+738.964053324" watchObservedRunningTime="2026-10-29 23:26:00.860839819 +0000 UTC m=+738.965100204"
Oct 29 23:26:00.907888 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:00.907828    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2" podStartSLOduration=1.7054519 podStartE2EDuration="1.907814515s" totalImagesPullingTime="202.362615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:00.90700372 +0000 UTC m=+739.011264106" watchObservedRunningTime="2026-10-29 23:26:00.907814515 +0000 UTC m=+739.012074900"
Oct 29 23:26:01.110449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:01.110418    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:26:01.180816 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:01.180786    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:26:02.665391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.665359    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"]
Oct 29 23:26:02.668752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.668728    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.692501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.692469    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"]
Oct 29 23:26:02.706117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.706086    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 29 23:26:02.744447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.744414    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.744588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.744461    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.744588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.744484    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8aa778-5206-47f8-8379-03976219cdcc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.744588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.744566    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptznh\" (UniqueName: \"kubernetes.io/projected/1e8aa778-5206-47f8-8379-03976219cdcc-kube-api-access-ptznh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.744706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.744666    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.744706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.744693    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.845602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.845568    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptznh\" (UniqueName: \"kubernetes.io/projected/1e8aa778-5206-47f8-8379-03976219cdcc-kube-api-access-ptznh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.845791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.845615    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.845791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.845636    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.845791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.845685    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.845791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.845727    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.845791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.845766    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8aa778-5206-47f8-8379-03976219cdcc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.846122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.846090    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.846290 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.846120    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.846290 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.846174    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.848244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.848216    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8aa778-5206-47f8-8379-03976219cdcc-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.848618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.848595    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8aa778-5206-47f8-8379-03976219cdcc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.871304 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.871272    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptznh\" (UniqueName: \"kubernetes.io/projected/1e8aa778-5206-47f8-8379-03976219cdcc-kube-api-access-ptznh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk\" (UID: \"1e8aa778-5206-47f8-8379-03976219cdcc\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:02.981205 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:02.981167    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:03.279678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:03.279653    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"]
Oct 29 23:26:03.282021 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:03.281988    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e8aa778_5206_47f8_8379_03976219cdcc.slice/crio-693865f2a7883b792985bb8cb0879b0fc950ed24a4987b2d5eda33993a585e1d WatchSource:0}: Error finding container 693865f2a7883b792985bb8cb0879b0fc950ed24a4987b2d5eda33993a585e1d: Status 404 returned error can't find the container with id 693865f2a7883b792985bb8cb0879b0fc950ed24a4987b2d5eda33993a585e1d
Oct 29 23:26:03.828791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:03.828754    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" event={"ID":"1e8aa778-5206-47f8-8379-03976219cdcc","Type":"ContainerStarted","Data":"74623f71e7aed6762433f15f8c3dcbbb64f7a23d13bb5319636bc5b57d90bec8"}
Oct 29 23:26:03.829210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:03.828804    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" event={"ID":"1e8aa778-5206-47f8-8379-03976219cdcc","Type":"ContainerStarted","Data":"693865f2a7883b792985bb8cb0879b0fc950ed24a4987b2d5eda33993a585e1d"}
Oct 29 23:26:03.881438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:03.881382    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podStartSLOduration=1.704563217 podStartE2EDuration="1.881366119s" totalImagesPullingTime="176.802902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:03.880522494 +0000 UTC m=+741.984782880" watchObservedRunningTime="2026-10-29 23:26:03.881366119 +0000 UTC m=+741.985626504"
Oct 29 23:26:03.982078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:03.982044    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:06.029690 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.029657    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"]
Oct 29 23:26:06.033314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.033292    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.050920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.050899    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 29 23:26:06.066003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.065978    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"]
Oct 29 23:26:06.074925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.074902    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.075034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.074951    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.075034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.074998    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c660259-6736-4175-a257-4e65889e0761-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.075034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.075016    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswr5\" (UniqueName: \"kubernetes.io/projected/5c660259-6736-4175-a257-4e65889e0761-kube-api-access-nswr5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.075176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.075044    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.075256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.075233    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.176316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.176283    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c660259-6736-4175-a257-4e65889e0761-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.176316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.176319    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswr5\" (UniqueName: \"kubernetes.io/projected/5c660259-6736-4175-a257-4e65889e0761-kube-api-access-nswr5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.176535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.176350    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.176635 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.176613    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.176716 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.176698    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.176792 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.176779    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.177023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.177001    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.177135 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.177018    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.177231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.177202    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.178623 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.178601    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c660259-6736-4175-a257-4e65889e0761-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.179327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.179304    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c660259-6736-4175-a257-4e65889e0761-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.205691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.205653    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswr5\" (UniqueName: \"kubernetes.io/projected/5c660259-6736-4175-a257-4e65889e0761-kube-api-access-nswr5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mbwdj\" (UID: \"5c660259-6736-4175-a257-4e65889e0761\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.346307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.346215    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:06.645627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.645550    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"]
Oct 29 23:26:06.649041 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:06.649013    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c660259_6736_4175_a257_4e65889e0761.slice/crio-a20bc71d6a3c5d08e2171e3e5f3da7ff430599469dff9cd16ea805e790c08d8f WatchSource:0}: Error finding container a20bc71d6a3c5d08e2171e3e5f3da7ff430599469dff9cd16ea805e790c08d8f: Status 404 returned error can't find the container with id a20bc71d6a3c5d08e2171e3e5f3da7ff430599469dff9cd16ea805e790c08d8f
Oct 29 23:26:06.849281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:06.849236    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj" event={"ID":"5c660259-6736-4175-a257-4e65889e0761","Type":"ContainerStarted","Data":"a20bc71d6a3c5d08e2171e3e5f3da7ff430599469dff9cd16ea805e790c08d8f"}
Oct 29 23:26:07.855438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:07.855399    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj" event={"ID":"5c660259-6736-4175-a257-4e65889e0761","Type":"ContainerStarted","Data":"c0924533df6ed06d1c903f05c3053839e49566ee1d6e37a65b7127a7d7121202"}
Oct 29 23:26:07.903132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:07.903086    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj" podStartSLOduration=2.6915771939999997 podStartE2EDuration="2.903073428s" totalImagesPullingTime="211.496234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:07.902230549 +0000 UTC m=+746.006490937" watchObservedRunningTime="2026-10-29 23:26:07.903073428 +0000 UTC m=+746.007333814"
Oct 29 23:26:08.180249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:08.180213    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:26:08.192626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:08.192596    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:26:08.346852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:08.346822    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:08.870830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:08.870804    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6"
Oct 29 23:26:09.924561 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:26:09.924511    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:26:10.112320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:10.112283    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:26:10.147266 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:10.147234    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:26:10.884762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:10.884729    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2"
Oct 29 23:26:12.982183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:12.982150    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:12.994762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:12.994730    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:13.898478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:13.898447    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:26:16.346457 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:16.346416    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:16.359120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:16.359089    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:16.908567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:16.908538    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj"
Oct 29 23:26:19.320117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.320078    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c8556598f-hglq5"]
Oct 29 23:26:19.323689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.323666    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.343626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.343604    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c8556598f-hglq5"]
Oct 29 23:26:19.425650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.425612    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d144a1e-9b31-4af5-8b4f-47f1fab87d4e-tls-cert\") pod \"authorino-6c8556598f-hglq5\" (UID: \"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e\") " pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.425794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.425750    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkx9r\" (UniqueName: \"kubernetes.io/projected/7d144a1e-9b31-4af5-8b4f-47f1fab87d4e-kube-api-access-hkx9r\") pod \"authorino-6c8556598f-hglq5\" (UID: \"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e\") " pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.526530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.526492    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d144a1e-9b31-4af5-8b4f-47f1fab87d4e-tls-cert\") pod \"authorino-6c8556598f-hglq5\" (UID: \"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e\") " pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.526672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.526582    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkx9r\" (UniqueName: \"kubernetes.io/projected/7d144a1e-9b31-4af5-8b4f-47f1fab87d4e-kube-api-access-hkx9r\") pod \"authorino-6c8556598f-hglq5\" (UID: \"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e\") " pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.528910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.528870    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d144a1e-9b31-4af5-8b4f-47f1fab87d4e-tls-cert\") pod \"authorino-6c8556598f-hglq5\" (UID: \"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e\") " pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.552319 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.552287    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkx9r\" (UniqueName: \"kubernetes.io/projected/7d144a1e-9b31-4af5-8b4f-47f1fab87d4e-kube-api-access-hkx9r\") pod \"authorino-6c8556598f-hglq5\" (UID: \"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e\") " pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.635684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.635601    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c8556598f-hglq5"
Oct 29 23:26:19.935558 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:19.935534    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c8556598f-hglq5"]
Oct 29 23:26:19.937864 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:19.937837    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d144a1e_9b31_4af5_8b4f_47f1fab87d4e.slice/crio-6b7d2bcfd1f3ea1723a5177470e872d2595b545b7d5c15ae68985e4ea6d8be39 WatchSource:0}: Error finding container 6b7d2bcfd1f3ea1723a5177470e872d2595b545b7d5c15ae68985e4ea6d8be39: Status 404 returned error can't find the container with id 6b7d2bcfd1f3ea1723a5177470e872d2595b545b7d5c15ae68985e4ea6d8be39
Oct 29 23:26:20.914468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:20.914376    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c8556598f-hglq5" event={"ID":"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e","Type":"ContainerStarted","Data":"366c33cc0c26383b34d27b430e879e0df13182bc89cbe6f1546149f9424c1f1b"}
Oct 29 23:26:20.914468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:20.914430    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c8556598f-hglq5" event={"ID":"7d144a1e-9b31-4af5-8b4f-47f1fab87d4e","Type":"ContainerStarted","Data":"6b7d2bcfd1f3ea1723a5177470e872d2595b545b7d5c15ae68985e4ea6d8be39"}
Oct 29 23:26:20.959261 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:20.959208    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6c8556598f-hglq5" podStartSLOduration=1.259261395 podStartE2EDuration="1.959194599s" totalImagesPullingTime="699.933204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:20.959079044 +0000 UTC m=+759.063339412" watchObservedRunningTime="2026-10-29 23:26:20.959194599 +0000 UTC m=+759.063454978"
Oct 29 23:26:20.985648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:20.985616    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-576ff54d58-ms2ph"]
Oct 29 23:26:20.985839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:20.985815    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-576ff54d58-ms2ph" podUID="b72e167c-2336-4b08-aeb8-ba505de13466" containerName="authorino" containerID="cri-o://783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4" gracePeriod=30
Oct 29 23:26:21.239905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.239863    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:26:21.343525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.343487    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72e167c-2336-4b08-aeb8-ba505de13466-tls-cert\" (UniqueName: \"kubernetes.io/secret/b72e167c-2336-4b08-aeb8-ba505de13466-tls-cert\") pod \"b72e167c-2336-4b08-aeb8-ba505de13466\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") "
Oct 29 23:26:21.343694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.343581    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72e167c-2336-4b08-aeb8-ba505de13466-kube-api-access-ngq8s\" (UniqueName: \"kubernetes.io/projected/b72e167c-2336-4b08-aeb8-ba505de13466-kube-api-access-ngq8s\") pod \"b72e167c-2336-4b08-aeb8-ba505de13466\" (UID: \"b72e167c-2336-4b08-aeb8-ba505de13466\") "
Oct 29 23:26:21.345567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.345535    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72e167c-2336-4b08-aeb8-ba505de13466-kube-api-access-ngq8s" pod "b72e167c-2336-4b08-aeb8-ba505de13466" (UID: "b72e167c-2336-4b08-aeb8-ba505de13466"). InnerVolumeSpecName "kube-api-access-ngq8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:26:21.353543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.353514    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72e167c-2336-4b08-aeb8-ba505de13466-tls-cert" pod "b72e167c-2336-4b08-aeb8-ba505de13466" (UID: "b72e167c-2336-4b08-aeb8-ba505de13466"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:26:21.445124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.445078    6203 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b72e167c-2336-4b08-aeb8-ba505de13466-tls-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:26:21.445124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.445110    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngq8s\" (UniqueName: \"kubernetes.io/projected/b72e167c-2336-4b08-aeb8-ba505de13466-kube-api-access-ngq8s\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:26:21.920428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.920390    6203 generic.go:356] "Generic (PLEG): container finished" podID="b72e167c-2336-4b08-aeb8-ba505de13466" containerID="783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4" exitCode=0
Oct 29 23:26:21.920934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.920452    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576ff54d58-ms2ph"
Oct 29 23:26:21.920934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.920508    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576ff54d58-ms2ph" event={"ID":"b72e167c-2336-4b08-aeb8-ba505de13466","Type":"ContainerDied","Data":"783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4"}
Oct 29 23:26:21.920934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.920541    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576ff54d58-ms2ph" event={"ID":"b72e167c-2336-4b08-aeb8-ba505de13466","Type":"ContainerDied","Data":"259b4eb23e6cdcc631ec3a6723f6e66f22e25196a91a6275846ba738610fa214"}
Oct 29 23:26:21.920934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.920563    6203 scope.go:122] "RemoveContainer" containerID="783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4"
Oct 29 23:26:21.929960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.929940    6203 scope.go:122] "RemoveContainer" containerID="783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4"
Oct 29 23:26:21.930259 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:26:21.930240    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4\": container with ID starting with 783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4 not found: ID does not exist" containerID="783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4"
Oct 29 23:26:21.930308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.930269    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4"} err="failed to get container status \"783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4\": rpc error: code = NotFound desc = could not find container \"783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4\": container with ID starting with 783216e7ede9608bc394b456e9467453bc6178415b1d7896f35c905c56befad4 not found: ID does not exist"
Oct 29 23:26:21.984573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.984537    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-576ff54d58-ms2ph"]
Oct 29 23:26:21.999280 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:21.999254    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-576ff54d58-ms2ph"]
Oct 29 23:26:22.017939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:22.017906    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72e167c-2336-4b08-aeb8-ba505de13466" path="/var/lib/kubelet/pods/b72e167c-2336-4b08-aeb8-ba505de13466/volumes"
Oct 29 23:26:34.776276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.776243    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-569f644b7c-nl8kw"]
Oct 29 23:26:34.776793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.776586    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b72e167c-2336-4b08-aeb8-ba505de13466" containerName="authorino"
Oct 29 23:26:34.776793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.776597    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72e167c-2336-4b08-aeb8-ba505de13466" containerName="authorino"
Oct 29 23:26:34.776793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.776669    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72e167c-2336-4b08-aeb8-ba505de13466" containerName="authorino"
Oct 29 23:26:34.782276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.782255    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:34.815658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.815617    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 29 23:26:34.815835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.815664    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nm8nz\""
Oct 29 23:26:34.815835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.815718    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 29 23:26:34.816583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.816558    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-569f644b7c-nl8kw"]
Oct 29 23:26:34.975289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.975247    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllpp\" (UniqueName: \"kubernetes.io/projected/42e769b0-8c1a-401c-8255-38bc5c4ba61d-kube-api-access-nllpp\") pod \"ai-gateway-controller-569f644b7c-nl8kw\" (UID: \"42e769b0-8c1a-401c-8255-38bc5c4ba61d\") " pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:34.975464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:34.975312    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e769b0-8c1a-401c-8255-38bc5c4ba61d-metrics-certs\") pod \"ai-gateway-controller-569f644b7c-nl8kw\" (UID: \"42e769b0-8c1a-401c-8255-38bc5c4ba61d\") " pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:35.076479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:35.076383    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e769b0-8c1a-401c-8255-38bc5c4ba61d-metrics-certs\") pod \"ai-gateway-controller-569f644b7c-nl8kw\" (UID: \"42e769b0-8c1a-401c-8255-38bc5c4ba61d\") " pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:35.076479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:35.076464    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllpp\" (UniqueName: \"kubernetes.io/projected/42e769b0-8c1a-401c-8255-38bc5c4ba61d-kube-api-access-nllpp\") pod \"ai-gateway-controller-569f644b7c-nl8kw\" (UID: \"42e769b0-8c1a-401c-8255-38bc5c4ba61d\") " pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:35.078864 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:35.078837    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e769b0-8c1a-401c-8255-38bc5c4ba61d-metrics-certs\") pod \"ai-gateway-controller-569f644b7c-nl8kw\" (UID: \"42e769b0-8c1a-401c-8255-38bc5c4ba61d\") " pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:35.103038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:35.103008    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllpp\" (UniqueName: \"kubernetes.io/projected/42e769b0-8c1a-401c-8255-38bc5c4ba61d-kube-api-access-nllpp\") pod \"ai-gateway-controller-569f644b7c-nl8kw\" (UID: \"42e769b0-8c1a-401c-8255-38bc5c4ba61d\") " pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:35.398894 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:35.398794    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:35.694895 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:35.694859    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-569f644b7c-nl8kw"]
Oct 29 23:26:35.696658 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:35.696634    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e769b0_8c1a_401c_8255_38bc5c4ba61d.slice/crio-952b690e030b99ba9564b7d1b28e265fb9ede5c68ff0bb1fd61fd4233404d789 WatchSource:0}: Error finding container 952b690e030b99ba9564b7d1b28e265fb9ede5c68ff0bb1fd61fd4233404d789: Status 404 returned error can't find the container with id 952b690e030b99ba9564b7d1b28e265fb9ede5c68ff0bb1fd61fd4233404d789
Oct 29 23:26:35.991553 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:35.991463    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw" event={"ID":"42e769b0-8c1a-401c-8255-38bc5c4ba61d","Type":"ContainerStarted","Data":"952b690e030b99ba9564b7d1b28e265fb9ede5c68ff0bb1fd61fd4233404d789"}
Oct 29 23:26:36.918621 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.918594    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"]
Oct 29 23:26:36.927459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.927433    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:36.938748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.938721    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"]
Oct 29 23:26:36.942900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.942866    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:36.952750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.952720    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 23:26:36.953778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.953727    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"]
Oct 29 23:26:36.962687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.962666    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f5fmz\""
Oct 29 23:26:36.970976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:36.970954    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"]
Oct 29 23:26:37.093488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093455    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.093488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093494    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093521    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093731    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093772    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093802    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwhw\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-kube-api-access-6zwhw\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093844    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcptn\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-kube-api-access-bcptn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093937    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.093979    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094010    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094046    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094166    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094191    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094214    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.094395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094262    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094307    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094363    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.094658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.094416    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.195691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.195602    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.195691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.195652    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.195691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.195682    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.196018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.195807    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.196018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.195862    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.196018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.195908    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.196018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.195982    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.196229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196029    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.196229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196091    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.196229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196205    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.196607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196250    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.196607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196360    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.196607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196373    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.196607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196423    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.196607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196442    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.196607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196466    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196647    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196747    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196829    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196969    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.196998    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.197030    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwhw\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-kube-api-access-6zwhw\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.197055    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcptn\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-kube-api-access-bcptn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.197067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.197055    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.197532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.197289    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.197532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.197349    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.197790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.197764    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.198064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.198042    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.198726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.198701    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.199913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.199864    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.200395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.200371    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.201478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.201458    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.223785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.223431    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.223785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.223746    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcptn\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-kube-api-access-bcptn\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.223969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.223792    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwhw\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-kube-api-access-6zwhw\") pod \"maas-default-gateway-openshift-default-746ff8f49f-qpzfx\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.223969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.223803    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-hjjpg\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.242833 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.242807    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:37.262947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.262854    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:37.565161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.565080    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"]
Oct 29 23:26:37.579617 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:37.579589    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1db0e3_77bd_47b0_aa83_3fc0cb08f77f.slice/crio-d8a2bac390ebc0b64d9a724404e1f2fbb11b14ea5c480bdc713a63c845125d61 WatchSource:0}: Error finding container d8a2bac390ebc0b64d9a724404e1f2fbb11b14ea5c480bdc713a63c845125d61: Status 404 returned error can't find the container with id d8a2bac390ebc0b64d9a724404e1f2fbb11b14ea5c480bdc713a63c845125d61
Oct 29 23:26:37.607338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:37.607267    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"]
Oct 29 23:26:37.624408 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:37.624380    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa18428_1dc5_4b21_87b1_f5860cc77eeb.slice/crio-f9498fb3fce0a64a8957751f8eb1cacc644f0c750af7ddbeb2409db18ca3bf3a WatchSource:0}: Error finding container f9498fb3fce0a64a8957751f8eb1cacc644f0c750af7ddbeb2409db18ca3bf3a: Status 404 returned error can't find the container with id f9498fb3fce0a64a8957751f8eb1cacc644f0c750af7ddbeb2409db18ca3bf3a
Oct 29 23:26:38.003015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:38.002986    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" event={"ID":"7fa18428-1dc5-4b21-87b1-f5860cc77eeb","Type":"ContainerStarted","Data":"f9498fb3fce0a64a8957751f8eb1cacc644f0c750af7ddbeb2409db18ca3bf3a"}
Oct 29 23:26:38.004654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:38.004623    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw" event={"ID":"42e769b0-8c1a-401c-8255-38bc5c4ba61d","Type":"ContainerStarted","Data":"23bb6c7ac5f225c23b4bd1818388e8a2359792039a32833732a0d47c5b29cc81"}
Oct 29 23:26:38.004804 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:38.004785    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:38.005907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:38.005866    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" event={"ID":"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f","Type":"ContainerStarted","Data":"d8a2bac390ebc0b64d9a724404e1f2fbb11b14ea5c480bdc713a63c845125d61"}
Oct 29 23:26:38.051854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:38.051805    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw" podStartSLOduration=1.8230089839999999 podStartE2EDuration="4.051793984s" totalImagesPullingTime="2.228785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:38.05124243 +0000 UTC m=+776.155502815" watchObservedRunningTime="2026-10-29 23:26:38.051793984 +0000 UTC m=+776.156054364"
Oct 29 23:26:39.928648 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:26:39.928601    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:26:40.493921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:40.493870    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:40.494023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:40.493974    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:40.494023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:40.494013    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:40.718489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:40.718456    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:40.718567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:40.718543    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:40.718609 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:40.718575    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:41.022154 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.022050    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" event={"ID":"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f","Type":"ContainerStarted","Data":"06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376"}
Oct 29 23:26:41.023546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.023524    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" event={"ID":"7fa18428-1dc5-4b21-87b1-f5860cc77eeb","Type":"ContainerStarted","Data":"c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2"}
Oct 29 23:26:41.161596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.161536    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podStartSLOduration=2.07080562 podStartE2EDuration="5.161520963s" totalImagesPullingTime="3.090715343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:41.160573778 +0000 UTC m=+779.264834163" watchObservedRunningTime="2026-10-29 23:26:41.161520963 +0000 UTC m=+779.265781342"
Oct 29 23:26:41.161794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.161773    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podStartSLOduration=2.25131181 podStartE2EDuration="5.161767791s" totalImagesPullingTime="2.910455981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:41.104741324 +0000 UTC m=+779.209001711" watchObservedRunningTime="2026-10-29 23:26:41.161767791 +0000 UTC m=+779.266028177"
Oct 29 23:26:41.245902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.244302    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:41.251906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.250027    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 23:26:41.251906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.250103    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 23:26:41.267903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.265498    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:41.274966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.273052    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:41.274966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:41.273137    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:42.245903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:42.243345    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 23:26:42.245903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:42.243425    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 23:26:42.265903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:42.264376    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:42.265903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:42.264444    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:43.250904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:43.246338    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 23:26:43.250904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:43.246404    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 23:26:43.273903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:43.266328    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:43.273903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:43.266410    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:44.250905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:44.247326    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 23:26:44.250905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:44.247391    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 23:26:44.267903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:44.266541    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:44.267903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:44.266604    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:45.250901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:45.249328    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 23:26:45.250901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:45.249393    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 23:26:45.267902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:45.265314    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:45.267902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:45.265378    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:46.250901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:46.250327    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 23:26:46.250901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:46.250396    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 23:26:46.272902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:46.266284    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:46.272902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:46.266347    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:47.243286 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:47.243253    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 29 23:26:47.243457 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:47.243308    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 29 23:26:47.263984 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:47.263949    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:47.264151 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:47.263993    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:48.264267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:48.264226    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-hjjpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 29 23:26:48.264680 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:48.264313    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 29 23:26:48.953321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:48.953282    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:48.954906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:48.954022    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:48.955041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:48.954991    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:26:49.013452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:49.013424    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-569f644b7c-nl8kw"
Oct 29 23:26:49.267334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:49.267253    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:49.267798 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:49.267659    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:49.268371 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:49.268350    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:26:51.997725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:51.997691    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"]
Oct 29 23:26:52.002905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.002866    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.045243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.045209    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"]
Oct 29 23:26:52.079776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.079734    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.079967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.079785    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.079967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.079864    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.079967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.079931    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.080097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.080019    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnk9\" (UniqueName: \"kubernetes.io/projected/9b2c909d-461e-4fff-9d3f-af4bc893adc0-kube-api-access-djnk9\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.080097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.080066    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.080164 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.080133    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.080196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.080174    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.080229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.080194    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181568    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181624    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181640    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181657    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181705    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181726    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181743    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.181778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181766    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.182161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.181793    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnk9\" (UniqueName: \"kubernetes.io/projected/9b2c909d-461e-4fff-9d3f-af4bc893adc0-kube-api-access-djnk9\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.182161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.182060    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.182161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.182137    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.182406 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.182382    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.182507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.182452    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.182565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.182545    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.184011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.183991    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.184303 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.184286    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.208071 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.208044    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2c909d-461e-4fff-9d3f-af4bc893adc0-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.208198 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.208099    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnk9\" (UniqueName: \"kubernetes.io/projected/9b2c909d-461e-4fff-9d3f-af4bc893adc0-kube-api-access-djnk9\") pod \"maas-default-gateway-openshift-default-746ff8f49f-rwn2k\" (UID: \"9b2c909d-461e-4fff-9d3f-af4bc893adc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.317791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.317703    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:52.646140 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.646112    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"]
Oct 29 23:26:52.647149 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:26:52.647123    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2c909d_461e_4fff_9d3f_af4bc893adc0.slice/crio-1ee460a3a5485be1c45620041ef118e85996eaa2976118b467d9c2755a8a832c WatchSource:0}: Error finding container 1ee460a3a5485be1c45620041ef118e85996eaa2976118b467d9c2755a8a832c: Status 404 returned error can't find the container with id 1ee460a3a5485be1c45620041ef118e85996eaa2976118b467d9c2755a8a832c
Oct 29 23:26:52.649405 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.649374    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:52.649463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.649438    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:52.649501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:52.649468    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:26:53.122160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:53.122126    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" event={"ID":"9b2c909d-461e-4fff-9d3f-af4bc893adc0","Type":"ContainerStarted","Data":"c4c0206d40beb46db5b96a0b7fb43331116d748efe171e21bb66f90cdeed40f6"}
Oct 29 23:26:53.122540 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:53.122185    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" event={"ID":"9b2c909d-461e-4fff-9d3f-af4bc893adc0","Type":"ContainerStarted","Data":"1ee460a3a5485be1c45620041ef118e85996eaa2976118b467d9c2755a8a832c"}
Oct 29 23:26:53.185455 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:53.185397    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podStartSLOduration=2.185378825 podStartE2EDuration="2.185378825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:26:53.185237688 +0000 UTC m=+791.289498074" watchObservedRunningTime="2026-10-29 23:26:53.185378825 +0000 UTC m=+791.289639212"
Oct 29 23:26:53.318498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:53.318469    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:26:53.319761 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:53.319736    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 23:26:53.319916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:53.319792    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 23:26:54.318549 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:54.318516    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 23:26:54.318976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:54.318567    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 23:26:55.318252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:55.318211    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 23:26:55.318431 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:55.318264    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 23:26:56.318540 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:56.318503    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 23:26:56.318971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:56.318561    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 23:26:57.318987 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:57.318949    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 23:26:57.319393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:57.319013    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 23:26:58.318799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:58.318764    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 23:26:58.319005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:58.318834    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 23:26:59.318831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:59.318793    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 29 23:26:59.319042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:26:59.318860    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 29 23:27:00.975343 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:00.975308    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:27:00.975819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:00.975672    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:27:00.976724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:00.976690    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k"
Oct 29 23:27:09.923529 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:27:09.923493    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:27:22.806218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.806183    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"]
Oct 29 23:27:22.813978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.813949    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.830046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.830014    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"]
Oct 29 23:27:22.842986 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.842957    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b978a0-openshift-default-dockercfg-jmbzw\""
Oct 29 23:27:22.875649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875614    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.875831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875664    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8036-da5c-4e3f-a7c1-bf149f46356b-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.875831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875703    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.875831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875729    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.875831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875756    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.875831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875791    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.876056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875841    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.876056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875898    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6wt\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-kube-api-access-6l6wt\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.876056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.875920    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.977431 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977391    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.977666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977454    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6wt\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-kube-api-access-6l6wt\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.977666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977491    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.977666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977560    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.977666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977585    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8036-da5c-4e3f-a7c1-bf149f46356b-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.977666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977620    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.977666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977647    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.978131 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977676    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.978131 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.977715    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.978300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.978271    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.978584 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.978565    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.978694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.978680    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.978852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.978831    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.979489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.979469    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8036-da5c-4e3f-a7c1-bf149f46356b-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.980492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.980463    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:22.980944 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:22.980925    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:23.005246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.005217    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:23.005912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.005856    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6wt\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-kube-api-access-6l6wt\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:23.151434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.151352    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:23.254709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.254679    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"]
Oct 29 23:27:23.257515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.257495    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.279681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.279659    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-53c44e-openshift-default-dockercfg-hgv77\""
Oct 29 23:27:23.279913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.279856    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.279985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.279965    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969fab72-22a9-498b-a688-284edd8f591e-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.280027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.280003    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.280027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.280019    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.280113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.280048    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.280203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.280188    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgng\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-kube-api-access-zkgng\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.280269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.280243    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.280347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.280330    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969fab72-22a9-498b-a688-284edd8f591e-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.280388 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.280376    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.285467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.285444    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"]
Oct 29 23:27:23.381685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381649    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.381685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381687    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.381978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381715    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.381978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381761    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgng\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-kube-api-access-zkgng\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.381978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381787    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.381978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381841    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969fab72-22a9-498b-a688-284edd8f591e-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.381978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381914    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.381978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.381977    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.382259 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.382028    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969fab72-22a9-498b-a688-284edd8f591e-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.382259 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.382211    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.382364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.382275    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.382364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.382328    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.382476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.382364    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.383271 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.383244    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969fab72-22a9-498b-a688-284edd8f591e-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.384533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.384475    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969fab72-22a9-498b-a688-284edd8f591e-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.384651 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.384533    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.423266 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.423239    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.432180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.432156    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgng\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-kube-api-access-zkgng\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.462829 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.462790    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"]
Oct 29 23:27:23.466021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.465998    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.473476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.473455    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"]
Oct 29 23:27:23.478953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.478913    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:23.479041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.479013    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:23.479082 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.479060    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:23.483035 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483009    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eeca893-2cf6-463a-b808-e49d9efa768d-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483062    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483090    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483191    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483310 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483281    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483332    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2k59\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-kube-api-access-c2k59\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483416 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483400    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483516    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.483649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.483592    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.493097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.493068    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"]
Oct 29 23:27:23.517616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.517584    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-aa7e96-openshift-default-dockercfg-8q4lr\""
Oct 29 23:27:23.576150 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.576115    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:23.584449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584422    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584605 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584464    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2k59\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-kube-api-access-c2k59\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584605 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584507    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584605 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584563    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584605    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584643    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eeca893-2cf6-463a-b808-e49d9efa768d-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584700    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584728    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.584765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584761    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.585045 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.584869    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.585154 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.585134    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.585473 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.585448    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eeca893-2cf6-463a-b808-e49d9efa768d-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.585756 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.585726    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.585983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.585960    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.587185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.587161    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.587474 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.587453    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.628733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.628697    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.630320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.630295    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2k59\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-kube-api-access-c2k59\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.783379 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.783299    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:23.912209 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.912184    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"]
Oct 29 23:27:23.914079 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:23.914054    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969fab72_22a9_498b_a688_284edd8f591e.slice/crio-1ff7ba732491c6eaaa41a3802ee2a9e6bb0698450c0808413d62cd5990d9a6a0 WatchSource:0}: Error finding container 1ff7ba732491c6eaaa41a3802ee2a9e6bb0698450c0808413d62cd5990d9a6a0: Status 404 returned error can't find the container with id 1ff7ba732491c6eaaa41a3802ee2a9e6bb0698450c0808413d62cd5990d9a6a0
Oct 29 23:27:23.916459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.916428    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:23.916538 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.916491    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:23.916538 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:23.916524    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:24.099967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.099830    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"]
Oct 29 23:27:24.102512 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:24.102477    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeca893_2cf6_463a_b808_e49d9efa768d.slice/crio-38b666d2a49c452c3d2eb3a5d5ea0265537397f2712017d55ca750faae3ccab0 WatchSource:0}: Error finding container 38b666d2a49c452c3d2eb3a5d5ea0265537397f2712017d55ca750faae3ccab0: Status 404 returned error can't find the container with id 38b666d2a49c452c3d2eb3a5d5ea0265537397f2712017d55ca750faae3ccab0
Oct 29 23:27:24.106999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.106393    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:24.106999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.106481    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:24.106999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.106521    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:24.284149 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.284108    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" event={"ID":"7eeca893-2cf6-463a-b808-e49d9efa768d","Type":"ContainerStarted","Data":"05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b"}
Oct 29 23:27:24.284357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.284175    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" event={"ID":"7eeca893-2cf6-463a-b808-e49d9efa768d","Type":"ContainerStarted","Data":"38b666d2a49c452c3d2eb3a5d5ea0265537397f2712017d55ca750faae3ccab0"}
Oct 29 23:27:24.285918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.285861    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" event={"ID":"969fab72-22a9-498b-a688-284edd8f591e","Type":"ContainerStarted","Data":"b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef"}
Oct 29 23:27:24.286054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.285943    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" event={"ID":"969fab72-22a9-498b-a688-284edd8f591e","Type":"ContainerStarted","Data":"1ff7ba732491c6eaaa41a3802ee2a9e6bb0698450c0808413d62cd5990d9a6a0"}
Oct 29 23:27:24.287574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.287550    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" event={"ID":"884c8036-da5c-4e3f-a7c1-bf149f46356b","Type":"ContainerStarted","Data":"be2e67945cae744f22f764c36b8ccf12d01bfe44bcc62ff255a9490968902026"}
Oct 29 23:27:24.287687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.287589    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" event={"ID":"884c8036-da5c-4e3f-a7c1-bf149f46356b","Type":"ContainerStarted","Data":"90a87d0dd9af98d4aab68bc4ac1fa29d566a081b78cc1e8be17ab8781f3033bb"}
Oct 29 23:27:24.342318 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.342263    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podStartSLOduration=1.3422493690000001 podStartE2EDuration="1.342249369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:24.341556182 +0000 UTC m=+822.445816568" watchObservedRunningTime="2026-10-29 23:27:24.342249369 +0000 UTC m=+822.446509751"
Oct 29 23:27:24.407509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.407412    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podStartSLOduration=1.407396781 podStartE2EDuration="1.407396781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:24.406802807 +0000 UTC m=+822.511063193" watchObservedRunningTime="2026-10-29 23:27:24.407396781 +0000 UTC m=+822.511657167"
Oct 29 23:27:24.461626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.461563    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" podStartSLOduration=2.461543741 podStartE2EDuration="2.461543741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:24.46048544 +0000 UTC m=+822.564745826" watchObservedRunningTime="2026-10-29 23:27:24.461543741 +0000 UTC m=+822.565804133"
Oct 29 23:27:24.577270 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.577235    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:24.598299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.597977    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:24.784269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:24.784237    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:25.152691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:25.151568    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:25.160835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:25.160806    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:25.292699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:25.292670    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:25.292936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:25.292719    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:25.293667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:25.293645    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:25.293782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:25.293765    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:25.789602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:25.789569    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:26.298044 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:26.298005    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:26.298754 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:26.298737    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:29.008016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.007976    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"]
Oct 29 23:27:29.012240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.012213    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.034640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.034612    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"]
Oct 29 23:27:29.041587 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.041559    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4q5\" (UniqueName: \"kubernetes.io/projected/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-kube-api-access-7f4q5\") pod \"payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.041731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.041602    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.047315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.047298    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-92f6b3\""
Oct 29 23:27:29.087433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.087401    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"]
Oct 29 23:27:29.090842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.090825    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.113188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.113157    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-92f6b3-dockercfg-nm67r\""
Oct 29 23:27:29.119527 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.119495    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"]
Oct 29 23:27:29.144503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.144466    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4q5\" (UniqueName: \"kubernetes.io/projected/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-kube-api-access-7f4q5\") pod \"payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.144805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.144778    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f22878-956b-4f01-a384-216eee25c2b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.145027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.145008    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.145182 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.145165    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf26l\" (UniqueName: \"kubernetes.io/projected/a9f22878-956b-4f01-a384-216eee25c2b8-kube-api-access-pf26l\") pod \"payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.146147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.146120    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.171341 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.171295    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4q5\" (UniqueName: \"kubernetes.io/projected/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-kube-api-access-7f4q5\") pod \"payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.246824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.246786    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf26l\" (UniqueName: \"kubernetes.io/projected/a9f22878-956b-4f01-a384-216eee25c2b8-kube-api-access-pf26l\") pod \"payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.247051 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.246917    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f22878-956b-4f01-a384-216eee25c2b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.247627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.247605    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f22878-956b-4f01-a384-216eee25c2b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.272753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.272678    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf26l\" (UniqueName: \"kubernetes.io/projected/a9f22878-956b-4f01-a384-216eee25c2b8-kube-api-access-pf26l\") pod \"payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.324781 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.324736    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:29.403317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.403227    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:29.629227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.629202    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"]
Oct 29 23:27:29.631446 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:29.631418    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9912d64_c0a0_4f89_90db_e4c43f5e14c5.slice/crio-f9488805fc53e86bea9e901b8be7a9955cc98e6fbcd383ab90e2b297ead551be WatchSource:0}: Error finding container f9488805fc53e86bea9e901b8be7a9955cc98e6fbcd383ab90e2b297ead551be: Status 404 returned error can't find the container with id f9488805fc53e86bea9e901b8be7a9955cc98e6fbcd383ab90e2b297ead551be
Oct 29 23:27:29.716028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:29.716000    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"]
Oct 29 23:27:29.717025 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:29.716994    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f22878_956b_4f01_a384_216eee25c2b8.slice/crio-fe9236fdacb489b6b66bd3a4924a6d810fb4d3ef74486e31652ede18af7fea9b WatchSource:0}: Error finding container fe9236fdacb489b6b66bd3a4924a6d810fb4d3ef74486e31652ede18af7fea9b: Status 404 returned error can't find the container with id fe9236fdacb489b6b66bd3a4924a6d810fb4d3ef74486e31652ede18af7fea9b
Oct 29 23:27:30.326939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.326898    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" event={"ID":"d9912d64-c0a0-4f89-90db-e4c43f5e14c5","Type":"ContainerStarted","Data":"f9488805fc53e86bea9e901b8be7a9955cc98e6fbcd383ab90e2b297ead551be"}
Oct 29 23:27:30.330763 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.330730    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" event={"ID":"a9f22878-956b-4f01-a384-216eee25c2b8","Type":"ContainerStarted","Data":"fe9236fdacb489b6b66bd3a4924a6d810fb4d3ef74486e31652ede18af7fea9b"}
Oct 29 23:27:30.690068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.690032    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"]
Oct 29 23:27:30.693839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.693804    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"]
Oct 29 23:27:30.694140 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.694113    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:30.698183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.698161    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:30.716611 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.716585    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-aa7e96\""
Oct 29 23:27:30.726225 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.720385    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"]
Oct 29 23:27:30.730963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.730937    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-aa7e96-dockercfg-gvtj8\""
Oct 29 23:27:30.732481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.732455    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"]
Oct 29 23:27:30.762830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.762796    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdd6c\" (UniqueName: \"kubernetes.io/projected/ee361d0d-7e9f-4d21-899b-0e360e7067ac-kube-api-access-gdd6c\") pod \"payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:30.762997 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.762845    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee361d0d-7e9f-4d21-899b-0e360e7067ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:30.763057 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.763005    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d39d18a-c0f4-4856-a880-e9bee6cdb224-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:30.763148 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.763114    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tmx\" (UniqueName: \"kubernetes.io/projected/3d39d18a-c0f4-4856-a880-e9bee6cdb224-kube-api-access-78tmx\") pod \"payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:30.864363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.864333    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tmx\" (UniqueName: \"kubernetes.io/projected/3d39d18a-c0f4-4856-a880-e9bee6cdb224-kube-api-access-78tmx\") pod \"payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:30.864587 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.864464    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdd6c\" (UniqueName: \"kubernetes.io/projected/ee361d0d-7e9f-4d21-899b-0e360e7067ac-kube-api-access-gdd6c\") pod \"payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:30.864587 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.864501    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee361d0d-7e9f-4d21-899b-0e360e7067ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:30.864587 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.864553    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d39d18a-c0f4-4856-a880-e9bee6cdb224-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:30.865461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.865436    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d39d18a-c0f4-4856-a880-e9bee6cdb224-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:30.865592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.865561    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee361d0d-7e9f-4d21-899b-0e360e7067ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:30.908038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.907933    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tmx\" (UniqueName: \"kubernetes.io/projected/3d39d18a-c0f4-4856-a880-e9bee6cdb224-kube-api-access-78tmx\") pod \"payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:30.908038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:30.907993    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdd6c\" (UniqueName: \"kubernetes.io/projected/ee361d0d-7e9f-4d21-899b-0e360e7067ac-kube-api-access-gdd6c\") pod \"payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:31.015348 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.015277    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:31.025222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.025025    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:31.340400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.340266    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"]
Oct 29 23:27:31.344436 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:31.344027    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee361d0d_7e9f_4d21_899b_0e360e7067ac.slice/crio-983fed6bed7d47f0c5427ed8fdcba10712d7afdf383ea307e9f0d9925a118989 WatchSource:0}: Error finding container 983fed6bed7d47f0c5427ed8fdcba10712d7afdf383ea307e9f0d9925a118989: Status 404 returned error can't find the container with id 983fed6bed7d47f0c5427ed8fdcba10712d7afdf383ea307e9f0d9925a118989
Oct 29 23:27:31.362296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.362211    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"]
Oct 29 23:27:31.374770 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:31.374735    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d39d18a_c0f4_4856_a880_e9bee6cdb224.slice/crio-0922f16bb892abac2fb86235f36338b90b505ee6a467ce4dda791e2b8333bcb8 WatchSource:0}: Error finding container 0922f16bb892abac2fb86235f36338b90b505ee6a467ce4dda791e2b8333bcb8: Status 404 returned error can't find the container with id 0922f16bb892abac2fb86235f36338b90b505ee6a467ce4dda791e2b8333bcb8
Oct 29 23:27:31.743238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.743206    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"]
Oct 29 23:27:31.747492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.747449    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:31.770282 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.770247    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-53c44e\""
Oct 29 23:27:31.770480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.770463    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-53c44e-dockercfg-7rrkq\""
Oct 29 23:27:31.774419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.774383    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nshhg\" (UniqueName: \"kubernetes.io/projected/0898b118-c645-49fc-b269-7884fb71d335-kube-api-access-nshhg\") pod \"payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:31.774580 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.774454    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0898b118-c645-49fc-b269-7884fb71d335-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:31.775027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.775004    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"]
Oct 29 23:27:31.805858 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.805822    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"]
Oct 29 23:27:31.810327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.810297    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:31.831602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.831572    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"]
Oct 29 23:27:31.876268 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.876188    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88c1e53-6f17-4c53-8b54-fb9880f29b88-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:31.876521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.876279    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nshhg\" (UniqueName: \"kubernetes.io/projected/0898b118-c645-49fc-b269-7884fb71d335-kube-api-access-nshhg\") pod \"payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:31.876521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.876339    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0898b118-c645-49fc-b269-7884fb71d335-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:31.876521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.876376    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmlxf\" (UniqueName: \"kubernetes.io/projected/c88c1e53-6f17-4c53-8b54-fb9880f29b88-kube-api-access-gmlxf\") pod \"payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:31.877515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.877482    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0898b118-c645-49fc-b269-7884fb71d335-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:31.922094 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.922031    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nshhg\" (UniqueName: \"kubernetes.io/projected/0898b118-c645-49fc-b269-7884fb71d335-kube-api-access-nshhg\") pod \"payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:31.977458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.977276    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmlxf\" (UniqueName: \"kubernetes.io/projected/c88c1e53-6f17-4c53-8b54-fb9880f29b88-kube-api-access-gmlxf\") pod \"payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:31.977458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.977418    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88c1e53-6f17-4c53-8b54-fb9880f29b88-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:31.978572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:31.978543    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88c1e53-6f17-4c53-8b54-fb9880f29b88-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:32.005275 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.005200    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmlxf\" (UniqueName: \"kubernetes.io/projected/c88c1e53-6f17-4c53-8b54-fb9880f29b88-kube-api-access-gmlxf\") pod \"payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:32.069633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.069329    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:32.131645 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.131359    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:32.363691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.363441    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" event={"ID":"3d39d18a-c0f4-4856-a880-e9bee6cdb224","Type":"ContainerStarted","Data":"0922f16bb892abac2fb86235f36338b90b505ee6a467ce4dda791e2b8333bcb8"}
Oct 29 23:27:32.366284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.366172    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" event={"ID":"ee361d0d-7e9f-4d21-899b-0e360e7067ac","Type":"ContainerStarted","Data":"983fed6bed7d47f0c5427ed8fdcba10712d7afdf383ea307e9f0d9925a118989"}
Oct 29 23:27:32.416078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.415971    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"]
Oct 29 23:27:32.424118 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:32.424081    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0898b118_c645_49fc_b269_7884fb71d335.slice/crio-af129ec732e996b1323fbd9399a2e4e5d1bc7d386929f23f7dc00c3014acabc7 WatchSource:0}: Error finding container af129ec732e996b1323fbd9399a2e4e5d1bc7d386929f23f7dc00c3014acabc7: Status 404 returned error can't find the container with id af129ec732e996b1323fbd9399a2e4e5d1bc7d386929f23f7dc00c3014acabc7
Oct 29 23:27:32.467531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.467487    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"]
Oct 29 23:27:32.475656 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:32.475611    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88c1e53_6f17_4c53_8b54_fb9880f29b88.slice/crio-caefc3727a1529d8bd36caeaf8d7051072f89d22337cc07c11055e6fdb93e587 WatchSource:0}: Error finding container caefc3727a1529d8bd36caeaf8d7051072f89d22337cc07c11055e6fdb93e587: Status 404 returned error can't find the container with id caefc3727a1529d8bd36caeaf8d7051072f89d22337cc07c11055e6fdb93e587
Oct 29 23:27:32.507062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.507018    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"]
Oct 29 23:27:32.511298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.511272    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.538504 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.538472    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"]
Oct 29 23:27:32.543935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.543886    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b978a0\""
Oct 29 23:27:32.589357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.586304    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e861dc13-1a62-4d01-94c4-e6e3288416de-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.589357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.586372    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wdlr\" (UniqueName: \"kubernetes.io/projected/e861dc13-1a62-4d01-94c4-e6e3288416de-kube-api-access-6wdlr\") pod \"payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.595050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.593103    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"]
Oct 29 23:27:32.598787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.598755    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:32.621351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.621278    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b978a0-dockercfg-ldqwr\""
Oct 29 23:27:32.622084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.622060    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"]
Oct 29 23:27:32.698865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.697009    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjrjw\" (UniqueName: \"kubernetes.io/projected/65ffd545-fdf2-43d3-a726-a6fe4099d02e-kube-api-access-qjrjw\") pod \"payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:32.698865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.697126    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e861dc13-1a62-4d01-94c4-e6e3288416de-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.698865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.697162    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ffd545-fdf2-43d3-a726-a6fe4099d02e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:32.698865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.697201    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdlr\" (UniqueName: \"kubernetes.io/projected/e861dc13-1a62-4d01-94c4-e6e3288416de-kube-api-access-6wdlr\") pod \"payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.698865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.698479    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e861dc13-1a62-4d01-94c4-e6e3288416de-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.728169 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.728135    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wdlr\" (UniqueName: \"kubernetes.io/projected/e861dc13-1a62-4d01-94c4-e6e3288416de-kube-api-access-6wdlr\") pod \"payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.799153 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.797938    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ffd545-fdf2-43d3-a726-a6fe4099d02e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:32.799153 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.798031    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjrjw\" (UniqueName: \"kubernetes.io/projected/65ffd545-fdf2-43d3-a726-a6fe4099d02e-kube-api-access-qjrjw\") pod \"payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:32.799153 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.799105    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ffd545-fdf2-43d3-a726-a6fe4099d02e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:32.825559 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.825531    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjrjw\" (UniqueName: \"kubernetes.io/projected/65ffd545-fdf2-43d3-a726-a6fe4099d02e-kube-api-access-qjrjw\") pod \"payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:32.832058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.832023    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:32.927468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:32.927428    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:33.169917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.169890    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"]
Oct 29 23:27:33.171842 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:33.171809    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode861dc13_1a62_4d01_94c4_e6e3288416de.slice/crio-aa7e6d15817f04dc4c8b8769933e8fdc8f6b5f5e3a0efc3efbad4440851a09c3 WatchSource:0}: Error finding container aa7e6d15817f04dc4c8b8769933e8fdc8f6b5f5e3a0efc3efbad4440851a09c3: Status 404 returned error can't find the container with id aa7e6d15817f04dc4c8b8769933e8fdc8f6b5f5e3a0efc3efbad4440851a09c3
Oct 29 23:27:33.259918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.259871    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"]
Oct 29 23:27:33.261398 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:33.261371    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ffd545_fdf2_43d3_a726_a6fe4099d02e.slice/crio-d26aa214a9e26d2bdc336d827f4a4483ab79dafcb7cc3b5ae9f83eb4cbe5b72c WatchSource:0}: Error finding container d26aa214a9e26d2bdc336d827f4a4483ab79dafcb7cc3b5ae9f83eb4cbe5b72c: Status 404 returned error can't find the container with id d26aa214a9e26d2bdc336d827f4a4483ab79dafcb7cc3b5ae9f83eb4cbe5b72c
Oct 29 23:27:33.374078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.374048    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" event={"ID":"65ffd545-fdf2-43d3-a726-a6fe4099d02e","Type":"ContainerStarted","Data":"30129b99ffea6d63c32d481a9ce4e7d7ba2aa6204cb3ecd319fa9bc31971e2b5"}
Oct 29 23:27:33.374509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.374099    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" event={"ID":"65ffd545-fdf2-43d3-a726-a6fe4099d02e","Type":"ContainerStarted","Data":"d26aa214a9e26d2bdc336d827f4a4483ab79dafcb7cc3b5ae9f83eb4cbe5b72c"}
Oct 29 23:27:33.374509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.374134    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:33.375793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.375768    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" event={"ID":"d9912d64-c0a0-4f89-90db-e4c43f5e14c5","Type":"ContainerStarted","Data":"7eff1d0d9574c407b855ec8edd4e8b668205ff569e150fca72b71f7a6dd8615d"}
Oct 29 23:27:33.375925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.375811    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:33.377377 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.377352    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" event={"ID":"a9f22878-956b-4f01-a384-216eee25c2b8","Type":"ContainerStarted","Data":"3174a0d55d2ae1a296258d0ef078003e4db701ae8325aee8abd0cb874c356552"}
Oct 29 23:27:33.377495 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.377464    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:33.378916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.378888    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" event={"ID":"3d39d18a-c0f4-4856-a880-e9bee6cdb224","Type":"ContainerStarted","Data":"ea24aa9b1d08811ec13822fdd7f976bcf6babb3535d1588a549d6c111ebeea2c"}
Oct 29 23:27:33.379014 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.378949    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:33.380430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.380407    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" event={"ID":"ee361d0d-7e9f-4d21-899b-0e360e7067ac","Type":"ContainerStarted","Data":"594c67051364ba4aa37a53551541810c2f10a24487e33b61a1e2b8573085e88b"}
Oct 29 23:27:33.380532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.380485    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:33.382106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.382083    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" event={"ID":"c88c1e53-6f17-4c53-8b54-fb9880f29b88","Type":"ContainerStarted","Data":"6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a"}
Oct 29 23:27:33.382242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.382127    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" event={"ID":"c88c1e53-6f17-4c53-8b54-fb9880f29b88","Type":"ContainerStarted","Data":"caefc3727a1529d8bd36caeaf8d7051072f89d22337cc07c11055e6fdb93e587"}
Oct 29 23:27:33.382242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.382158    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:33.383617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.383597    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" event={"ID":"e861dc13-1a62-4d01-94c4-e6e3288416de","Type":"ContainerStarted","Data":"f8c174fb0a1ec584c1a6dc060330e42ebec8b9ccb87abee027dbeb05e333e8ea"}
Oct 29 23:27:33.383694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.383637    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:33.383694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.383663    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" event={"ID":"e861dc13-1a62-4d01-94c4-e6e3288416de","Type":"ContainerStarted","Data":"aa7e6d15817f04dc4c8b8769933e8fdc8f6b5f5e3a0efc3efbad4440851a09c3"}
Oct 29 23:27:33.385095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.385075    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" event={"ID":"0898b118-c645-49fc-b269-7884fb71d335","Type":"ContainerStarted","Data":"f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1"}
Oct 29 23:27:33.385176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.385107    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" event={"ID":"0898b118-c645-49fc-b269-7884fb71d335","Type":"ContainerStarted","Data":"af129ec732e996b1323fbd9399a2e4e5d1bc7d386929f23f7dc00c3014acabc7"}
Oct 29 23:27:33.385231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.385218    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:33.421031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.420938    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" podStartSLOduration=1.420924171 podStartE2EDuration="1.420924171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.418817467 +0000 UTC m=+831.523077853" watchObservedRunningTime="2026-10-29 23:27:33.420924171 +0000 UTC m=+831.525184556"
Oct 29 23:27:33.465556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.465505    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" podStartSLOduration=1.471548142 podStartE2EDuration="4.465486713s" totalImagesPullingTime="2.993938571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.464174037 +0000 UTC m=+831.568434423" watchObservedRunningTime="2026-10-29 23:27:33.465486713 +0000 UTC m=+831.569747102"
Oct 29 23:27:33.563814 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.563756    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" podStartSLOduration=2.034001167 podStartE2EDuration="2.563742354s" totalImagesPullingTime="529.741187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.516537141 +0000 UTC m=+831.620797527" watchObservedRunningTime="2026-10-29 23:27:33.563742354 +0000 UTC m=+831.668002779"
Oct 29 23:27:33.564033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.563968    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" podStartSLOduration=1.622332487 podStartE2EDuration="3.563959188s" totalImagesPullingTime="1.941626701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.563405679 +0000 UTC m=+831.667666046" watchObservedRunningTime="2026-10-29 23:27:33.563959188 +0000 UTC m=+831.668219576"
Oct 29 23:27:33.612048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.611989    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" podStartSLOduration=1.690967151 podStartE2EDuration="3.61197518s" totalImagesPullingTime="1.921008029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.611251932 +0000 UTC m=+831.715512320" watchObservedRunningTime="2026-10-29 23:27:33.61197518 +0000 UTC m=+831.716235566"
Oct 29 23:27:33.657556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.657507    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" podStartSLOduration=2.302929565 podStartE2EDuration="2.65749392s" totalImagesPullingTime="354.564355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.656727045 +0000 UTC m=+831.760987432" watchObservedRunningTime="2026-10-29 23:27:33.65749392 +0000 UTC m=+831.761754305"
Oct 29 23:27:33.701871 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.701772    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" podStartSLOduration=1.701758795 podStartE2EDuration="1.701758795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.701397868 +0000 UTC m=+831.805658275" watchObservedRunningTime="2026-10-29 23:27:33.701758795 +0000 UTC m=+831.806019182"
Oct 29 23:27:33.747539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:33.747487    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" podStartSLOduration=2.826924613 podStartE2EDuration="5.747472349s" totalImagesPullingTime="2.920547736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:33.747029058 +0000 UTC m=+831.851289444" watchObservedRunningTime="2026-10-29 23:27:33.747472349 +0000 UTC m=+831.851732736"
Oct 29 23:27:35.854873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.853522    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"]
Oct 29 23:27:35.860571 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.860528    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:35.868984 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.868951    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"]
Oct 29 23:27:35.881357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.881321    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ae1364\""
Oct 29 23:27:35.881357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.881322    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ae1364-dockercfg-jrl7q\""
Oct 29 23:27:35.904972 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.904938    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"]
Oct 29 23:27:35.909062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.909033    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:35.957904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.929237    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"]
Oct 29 23:27:35.958774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.952507    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nprw\" (UniqueName: \"kubernetes.io/projected/702ac16e-58f7-402f-90e7-75d68204613a-kube-api-access-5nprw\") pod \"payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:35.958928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.958865    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:35.958971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.958955    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ac16e-58f7-402f-90e7-75d68204613a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:35.959028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:35.959022    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9js22\" (UniqueName: \"kubernetes.io/projected/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-kube-api-access-9js22\") pod \"payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:36.060473 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.060425    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nprw\" (UniqueName: \"kubernetes.io/projected/702ac16e-58f7-402f-90e7-75d68204613a-kube-api-access-5nprw\") pod \"payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:36.060656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.060499    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:36.060656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.060546    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ac16e-58f7-402f-90e7-75d68204613a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:36.060656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.060614    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9js22\" (UniqueName: \"kubernetes.io/projected/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-kube-api-access-9js22\") pod \"payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:36.061538 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.061508    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:36.061810 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.061782    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ac16e-58f7-402f-90e7-75d68204613a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:36.086299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.086269    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nprw\" (UniqueName: \"kubernetes.io/projected/702ac16e-58f7-402f-90e7-75d68204613a-kube-api-access-5nprw\") pod \"payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:36.086403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.086272    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9js22\" (UniqueName: \"kubernetes.io/projected/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-kube-api-access-9js22\") pod \"payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:36.178642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.178598    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:36.222567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.222528    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:36.505650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.505618    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"]
Oct 29 23:27:36.568504 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:36.568465    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"]
Oct 29 23:27:36.582111 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:36.582076    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702ac16e_58f7_402f_90e7_75d68204613a.slice/crio-5aa6f5a4e383732c104e8f1a861f5aba10e4a1d327fde01afbd1cd629d4d20a8 WatchSource:0}: Error finding container 5aa6f5a4e383732c104e8f1a861f5aba10e4a1d327fde01afbd1cd629d4d20a8: Status 404 returned error can't find the container with id 5aa6f5a4e383732c104e8f1a861f5aba10e4a1d327fde01afbd1cd629d4d20a8
Oct 29 23:27:37.412357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.412313    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" event={"ID":"702ac16e-58f7-402f-90e7-75d68204613a","Type":"ContainerStarted","Data":"8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66"}
Oct 29 23:27:37.412801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.412379    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" event={"ID":"702ac16e-58f7-402f-90e7-75d68204613a","Type":"ContainerStarted","Data":"5aa6f5a4e383732c104e8f1a861f5aba10e4a1d327fde01afbd1cd629d4d20a8"}
Oct 29 23:27:37.412801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.412543    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:37.413869 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.413848    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" event={"ID":"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873","Type":"ContainerStarted","Data":"48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11"}
Oct 29 23:27:37.413967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.413892    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" event={"ID":"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873","Type":"ContainerStarted","Data":"60816fadae7163980ac0a88f1cad09420a9b323c04c706373b1a79c0c72e226c"}
Oct 29 23:27:37.414025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.413985    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:37.420130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.420109    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"]
Oct 29 23:27:37.423532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.423518    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:37.458336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.456579    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-ae1784\""
Oct 29 23:27:37.458336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.456723    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-ae1784-dockercfg-hgdn9\""
Oct 29 23:27:37.458947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.458912    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"]
Oct 29 23:27:37.473231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.473177    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" podStartSLOduration=2.473164683 podStartE2EDuration="2.473164683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:37.471297567 +0000 UTC m=+835.575557953" watchObservedRunningTime="2026-10-29 23:27:37.473164683 +0000 UTC m=+835.577425068"
Oct 29 23:27:37.578673 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.578618    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebd8bf-7fff-47e9-b706-6baf7f325180-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:37.578831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.578778    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqqr\" (UniqueName: \"kubernetes.io/projected/71ebd8bf-7fff-47e9-b706-6baf7f325180-kube-api-access-5dqqr\") pod \"payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:37.586226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.586147    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" podStartSLOduration=2.586128214 podStartE2EDuration="2.586128214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:37.531249104 +0000 UTC m=+835.635509492" watchObservedRunningTime="2026-10-29 23:27:37.586128214 +0000 UTC m=+835.690388603"
Oct 29 23:27:37.679522 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.679478    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebd8bf-7fff-47e9-b706-6baf7f325180-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:37.679862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.679836    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqqr\" (UniqueName: \"kubernetes.io/projected/71ebd8bf-7fff-47e9-b706-6baf7f325180-kube-api-access-5dqqr\") pod \"payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:37.680783 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.680755    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebd8bf-7fff-47e9-b706-6baf7f325180-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:37.708218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.708190    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqqr\" (UniqueName: \"kubernetes.io/projected/71ebd8bf-7fff-47e9-b706-6baf7f325180-kube-api-access-5dqqr\") pod \"payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:37.766554 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:37.766521    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:38.064128 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:38.064102    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"]
Oct 29 23:27:38.065169 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:27:38.065142    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ebd8bf_7fff_47e9_b706_6baf7f325180.slice/crio-15d55c27fcd0559e7ef0d454410df2e94c33be3b66d46def0746a6462766e474 WatchSource:0}: Error finding container 15d55c27fcd0559e7ef0d454410df2e94c33be3b66d46def0746a6462766e474: Status 404 returned error can't find the container with id 15d55c27fcd0559e7ef0d454410df2e94c33be3b66d46def0746a6462766e474
Oct 29 23:27:38.419803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:38.419772    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" event={"ID":"71ebd8bf-7fff-47e9-b706-6baf7f325180","Type":"ContainerStarted","Data":"e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3"}
Oct 29 23:27:38.420312 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:38.419921    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:38.420312 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:38.419961    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" event={"ID":"71ebd8bf-7fff-47e9-b706-6baf7f325180","Type":"ContainerStarted","Data":"15d55c27fcd0559e7ef0d454410df2e94c33be3b66d46def0746a6462766e474"}
Oct 29 23:27:38.478542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:38.478482    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" podStartSLOduration=1.478467417 podStartE2EDuration="1.478467417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:27:38.476830558 +0000 UTC m=+836.581090943" watchObservedRunningTime="2026-10-29 23:27:38.478467417 +0000 UTC m=+836.582727838"
Oct 29 23:27:39.964232 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:27:39.964079    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:27:44.396529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.396490    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:27:44.397125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.396960    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:27:44.397302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.397279    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:27:44.397796 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.397769    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:27:44.398548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.398525    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:27:44.399622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.399600    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:27:44.399952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.399926    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:27:44.400036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:44.399984    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:27:46.487432 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:46.487399    6203 generic.go:356] "Generic (PLEG): container finished" podID="969fab72-22a9-498b-a688-284edd8f591e" containerID="b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef" exitCode=0
Oct 29 23:27:46.487982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:46.487468    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" event={"ID":"969fab72-22a9-498b-a688-284edd8f591e","Type":"ContainerDied","Data":"b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef"}
Oct 29 23:27:46.487982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:46.487848    6203 scope.go:122] "RemoveContainer" containerID="b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef"
Oct 29 23:27:46.489272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:46.489239    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:46.489387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:46.489307    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:46.489387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:46.489337    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:47.496392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:47.496337    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" event={"ID":"969fab72-22a9-498b-a688-284edd8f591e","Type":"ContainerStarted","Data":"9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3"}
Oct 29 23:27:47.583691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:47.583657    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:47.583910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:47.583798    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 23:27:47.583910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:47.583840    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 23:27:48.422398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:48.422310    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:27:48.422566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:48.422408    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:27:48.577333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:48.577292    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 23:27:48.577966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:48.577931    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 23:27:49.456216 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:49.455153    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:27:49.581421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:49.581384    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 23:27:49.582041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:49.581986    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 23:27:50.578401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:50.578333    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 23:27:50.578621 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:50.578403    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 23:27:51.578900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:51.578317    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 23:27:51.578900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:51.578398    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 23:27:52.527200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.527170    6203 generic.go:356] "Generic (PLEG): container finished" podID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerID="be2e67945cae744f22f764c36b8ccf12d01bfe44bcc62ff255a9490968902026" exitCode=0
Oct 29 23:27:52.527376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.527232    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" event={"ID":"884c8036-da5c-4e3f-a7c1-bf149f46356b","Type":"ContainerDied","Data":"be2e67945cae744f22f764c36b8ccf12d01bfe44bcc62ff255a9490968902026"}
Oct 29 23:27:52.527676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.527658    6203 scope.go:122] "RemoveContainer" containerID="be2e67945cae744f22f764c36b8ccf12d01bfe44bcc62ff255a9490968902026"
Oct 29 23:27:52.550900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.549934    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:52.550900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.550052    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:52.550900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.550111    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:52.576707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.576679    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 29 23:27:52.576841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:52.576735    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 29 23:27:53.550927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:53.550554    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" event={"ID":"884c8036-da5c-4e3f-a7c1-bf149f46356b","Type":"ContainerStarted","Data":"a35f185a27425db3d6d79f009c05481df89e94523dfc74119916513fe662c2d1"}
Oct 29 23:27:53.582630 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:53.582599    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:53.582949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:53.582922    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:53.583855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:53.583828    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:27:54.156905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:54.153079    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:54.156905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:54.153228    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 23:27:54.156905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:54.153275    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 23:27:55.153917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:55.153315    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 23:27:55.153917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:55.153388    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 23:27:55.293367 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:55.293334    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:56.153323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:56.153282    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 23:27:56.153528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:56.153346    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 23:27:56.914865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:56.913408    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:49620->10.134.0.38:15021: read: connection reset by peer" start-of-body=
Oct 29 23:27:56.914865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:56.913473    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:49620->10.134.0.38:15021: read: connection reset by peer"
Oct 29 23:27:57.153353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.153272    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 23:27:57.153353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.153339    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 23:27:57.580785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.580739    6203 generic.go:356] "Generic (PLEG): container finished" podID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerID="05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b" exitCode=0
Oct 29 23:27:57.581226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.580802    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" event={"ID":"7eeca893-2cf6-463a-b808-e49d9efa768d","Type":"ContainerDied","Data":"05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b"}
Oct 29 23:27:57.581372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.581352    6203 scope.go:122] "RemoveContainer" containerID="05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b"
Oct 29 23:27:57.582949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.582918    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:57.583049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.582985    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:57.583049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:57.583018    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:27:58.157338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:58.157298    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Oct 29 23:27:58.158014 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:58.157970    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Oct 29 23:27:58.595935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:58.595865    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" event={"ID":"7eeca893-2cf6-463a-b808-e49d9efa768d","Type":"ContainerStarted","Data":"bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744"}
Oct 29 23:27:58.783531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:58.783493    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:27:58.783740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:58.783715    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 23:27:58.783821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:58.783769    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 23:27:59.158028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:59.157999    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:59.159145 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:59.159124    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:27:59.784193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:59.784157    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 23:27:59.784370 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:27:59.784211    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 23:28:00.784022 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:00.783980    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 23:28:00.784513 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:00.784049    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 23:28:01.784051 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:01.784016    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 23:28:01.784499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:01.784089    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 23:28:02.783901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:02.783847    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 23:28:02.784093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:02.783933    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 23:28:03.789902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:03.783931    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 29 23:28:03.789902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:03.783994    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 29 23:28:04.787560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:04.787527    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:28:04.787806 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:04.787785    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:28:04.788694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:04.788663    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:28:09.177245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.177212    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"]
Oct 29 23:28:09.181787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.181767    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.220186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.220154    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b978a0\"/\"kube-root-ca.crt\""
Oct 29 23:28:09.220186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.220165    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b978a0\"/\"openshift-service-ca.crt\""
Oct 29 23:28:09.223081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.223056    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"]
Oct 29 23:28:09.235769 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.235737    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b978a0\"/\"default-dockercfg-zzj7v\""
Oct 29 23:28:09.235993 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.235978    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b978a0\"/\"simulator-w0-b978a0-kserve-self-signed-certs\""
Oct 29 23:28:09.287140 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.287103    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-tmp-dir\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.287325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.287154    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-model-cache\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.287325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.287263    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtv59\" (UniqueName: \"kubernetes.io/projected/73462746-78dd-4ec9-8d31-5e9fb54d7945-kube-api-access-gtv59\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.287325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.287303    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-home\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.287435 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.287331    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-dshm\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.287471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.287444    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73462746-78dd-4ec9-8d31-5e9fb54d7945-tls-certs\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.388363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388331    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73462746-78dd-4ec9-8d31-5e9fb54d7945-tls-certs\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.388524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388393    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-tmp-dir\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.388524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388433    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-model-cache\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.388524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388470    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtv59\" (UniqueName: \"kubernetes.io/projected/73462746-78dd-4ec9-8d31-5e9fb54d7945-kube-api-access-gtv59\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.388524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388518    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-home\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.388705 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388546    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-dshm\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.388960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388932    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-tmp-dir\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.389077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.388974    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-model-cache\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.389077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.389008    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-home\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.390747 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.390722    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-dshm\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.390855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.390843    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73462746-78dd-4ec9-8d31-5e9fb54d7945-tls-certs\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.414588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.414565    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtv59\" (UniqueName: \"kubernetes.io/projected/73462746-78dd-4ec9-8d31-5e9fb54d7945-kube-api-access-gtv59\") pod \"simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") " pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.496338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.496238    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:09.799298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:09.799271    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"]
Oct 29 23:28:09.801092 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:28:09.801064    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73462746_78dd_4ec9_8d31_5e9fb54d7945.slice/crio-49145fad01ed976a82e8d4037d52132e0ed36af903af91f9b2a711082ca3e7af WatchSource:0}: Error finding container 49145fad01ed976a82e8d4037d52132e0ed36af903af91f9b2a711082ca3e7af: Status 404 returned error can't find the container with id 49145fad01ed976a82e8d4037d52132e0ed36af903af91f9b2a711082ca3e7af
Oct 29 23:28:09.925404 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:28:09.925370    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:28:10.660239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:10.660204    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" event={"ID":"73462746-78dd-4ec9-8d31-5e9fb54d7945","Type":"ContainerStarted","Data":"e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4"}
Oct 29 23:28:10.660666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:10.660252    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" event={"ID":"73462746-78dd-4ec9-8d31-5e9fb54d7945","Type":"ContainerStarted","Data":"49145fad01ed976a82e8d4037d52132e0ed36af903af91f9b2a711082ca3e7af"}
Oct 29 23:28:10.758863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:10.758813    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podStartSLOduration=1.758799218 podStartE2EDuration="1.758799218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:10.757651292 +0000 UTC m=+868.861911677" watchObservedRunningTime="2026-10-29 23:28:10.758799218 +0000 UTC m=+868.863059603"
Oct 29 23:28:11.497194 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:11.497159    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:19.497433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:19.497394    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:19.510117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:19.510090    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:19.723293 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:19.723264    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:28:22.896557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:22.896517    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"]
Oct 29 23:28:22.901450 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:22.901429    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:22.925094 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:22.925065    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"]
Oct 29 23:28:22.937931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:22.937905    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ae1364\"/\"default-dockercfg-kphlj\""
Oct 29 23:28:22.937931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:22.937920    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ae1364\"/\"kube-root-ca.crt\""
Oct 29 23:28:22.937931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:22.937907    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ae1364\"/\"openshift-service-ca.crt\""
Oct 29 23:28:22.938122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:22.937975    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ae1364\"/\"simulator-w2-ae1364-kserve-self-signed-certs\""
Oct 29 23:28:23.030003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.029968    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjn5\" (UniqueName: \"kubernetes.io/projected/a5fcc36f-6010-4bea-874c-dcc395e52ff1-kube-api-access-cqjn5\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.030180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.030025    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-home\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.030180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.030078    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-model-cache\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.030180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.030120    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tmp-dir\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.030180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.030141    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tls-certs\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.030398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.030297    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-dshm\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.130965    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjn5\" (UniqueName: \"kubernetes.io/projected/a5fcc36f-6010-4bea-874c-dcc395e52ff1-kube-api-access-cqjn5\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131031    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-home\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131060    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-model-cache\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131120    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tmp-dir\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131148    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tls-certs\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131218    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-dshm\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131468    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-home\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131511    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-model-cache\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.131568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.131545    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tmp-dir\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.133387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.133357    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-dshm\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.133628 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.133610    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tls-certs\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.157281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.157213    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjn5\" (UniqueName: \"kubernetes.io/projected/a5fcc36f-6010-4bea-874c-dcc395e52ff1-kube-api-access-cqjn5\") pod \"simulator-w2-ae1364-kserve-dbd84cf97-6bxcv\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") " pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.215639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.215609    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:23.518515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.518485    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"]
Oct 29 23:28:23.520506 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:28:23.520478    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5fcc36f_6010_4bea_874c_dcc395e52ff1.slice/crio-b4ce7553d21b3cff0848c7308ab8bd3021b3a99af17fde4039fe61a606012ca8 WatchSource:0}: Error finding container b4ce7553d21b3cff0848c7308ab8bd3021b3a99af17fde4039fe61a606012ca8: Status 404 returned error can't find the container with id b4ce7553d21b3cff0848c7308ab8bd3021b3a99af17fde4039fe61a606012ca8
Oct 29 23:28:23.731967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.731924    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" event={"ID":"a5fcc36f-6010-4bea-874c-dcc395e52ff1","Type":"ContainerStarted","Data":"5b8d6766fbc885fde5bbd3b9457b3e8b713601a3dda001f8718f232e3d5530be"}
Oct 29 23:28:23.732159 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.731977    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" event={"ID":"a5fcc36f-6010-4bea-874c-dcc395e52ff1","Type":"ContainerStarted","Data":"b4ce7553d21b3cff0848c7308ab8bd3021b3a99af17fde4039fe61a606012ca8"}
Oct 29 23:28:23.809748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:23.809640    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podStartSLOduration=1.809625476 podStartE2EDuration="1.809625476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:23.806991226 +0000 UTC m=+881.911251612" watchObservedRunningTime="2026-10-29 23:28:23.809625476 +0000 UTC m=+881.913885861"
Oct 29 23:28:24.216146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:24.216111    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:29.279103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.279054    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"]
Oct 29 23:28:29.284324 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.284296    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.303469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.303444    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"]
Oct 29 23:28:29.316332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.316302    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-53c44e\"/\"kube-root-ca.crt\""
Oct 29 23:28:29.316486 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.316303    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-53c44e\"/\"openshift-service-ca.crt\""
Oct 29 23:28:29.316486 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.316346    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-53c44e\"/\"default-dockercfg-z6kgj\""
Oct 29 23:28:29.318968 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.318950    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-53c44e\"/\"premium-w5-53c44e-kserve-self-signed-certs\""
Oct 29 23:28:29.401390 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.401351    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-home\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.401560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.401415    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-model-cache\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.401607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.401556    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-tmp-dir\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.401607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.401588    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24382701-d877-4546-a484-3a88dcb16266-tls-certs\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.401683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.401655    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9jm\" (UniqueName: \"kubernetes.io/projected/24382701-d877-4546-a484-3a88dcb16266-kube-api-access-dg9jm\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.401727 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.401682    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-dshm\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.502579 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.502547    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-tmp-dir\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.502579 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.502580    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24382701-d877-4546-a484-3a88dcb16266-tls-certs\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.502801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.502623    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9jm\" (UniqueName: \"kubernetes.io/projected/24382701-d877-4546-a484-3a88dcb16266-kube-api-access-dg9jm\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.502801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.502650    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-dshm\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.502801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.502685    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-home\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.502801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.502741    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-model-cache\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.503084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.503057    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-tmp-dir\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.503240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.503216    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-model-cache\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.503299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.503229    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-home\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.504928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.504908    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-dshm\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.505446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.505427    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24382701-d877-4546-a484-3a88dcb16266-tls-certs\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.529059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.529032    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9jm\" (UniqueName: \"kubernetes.io/projected/24382701-d877-4546-a484-3a88dcb16266-kube-api-access-dg9jm\") pod \"premium-w5-53c44e-kserve-8466bfd89c-dsfgr\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") " pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.601939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.601816    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:29.901201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:29.901176    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"]
Oct 29 23:28:29.901979 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:28:29.901946    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24382701_d877_4546_a484_3a88dcb16266.slice/crio-c9ccd456ab6134c9606c33076a9eb45ae2bc272ac1ccf13072b89acc2a549645 WatchSource:0}: Error finding container c9ccd456ab6134c9606c33076a9eb45ae2bc272ac1ccf13072b89acc2a549645: Status 404 returned error can't find the container with id c9ccd456ab6134c9606c33076a9eb45ae2bc272ac1ccf13072b89acc2a549645
Oct 29 23:28:30.776310 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:30.776272    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" event={"ID":"24382701-d877-4546-a484-3a88dcb16266","Type":"ContainerStarted","Data":"c5e1d1d8275836daf060f1c95616f106cbfda0eb8d14934510d3f922e7e011ec"}
Oct 29 23:28:30.776722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:30.776327    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" event={"ID":"24382701-d877-4546-a484-3a88dcb16266","Type":"ContainerStarted","Data":"c9ccd456ab6134c9606c33076a9eb45ae2bc272ac1ccf13072b89acc2a549645"}
Oct 29 23:28:30.825308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:30.825222    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podStartSLOduration=1.82520254 podStartE2EDuration="1.82520254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:30.82259591 +0000 UTC m=+888.926856296" watchObservedRunningTime="2026-10-29 23:28:30.82520254 +0000 UTC m=+888.929462927"
Oct 29 23:28:31.602718 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:31.602687    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:33.216740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:33.216702    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:33.229480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:33.229458    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:33.803775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:33.803740    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:28:36.044279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.044235    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"]
Oct 29 23:28:36.050003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.049977    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.071601 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.071564    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"]
Oct 29 23:28:36.082604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.082579    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-92f6b3\"/\"default-dockercfg-dtsbz\""
Oct 29 23:28:36.082751 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.082682    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-92f6b3\"/\"premium-w3-92f6b3-kserve-self-signed-certs\""
Oct 29 23:28:36.089430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089403    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-dshm\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.089548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089431    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-model-cache\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.089548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089471    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34fc68-99bf-401e-a88f-087ddcdd077c-tls-certs\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.089548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089533    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-home\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.089727 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089700    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v584j\" (UniqueName: \"kubernetes.io/projected/5b34fc68-99bf-401e-a88f-087ddcdd077c-kube-api-access-v584j\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.089866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089755    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-tmp-dir\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.089866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089756    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-92f6b3\"/\"kube-root-ca.crt\""
Oct 29 23:28:36.089866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.089805    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-92f6b3\"/\"openshift-service-ca.crt\""
Oct 29 23:28:36.190568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.190534    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34fc68-99bf-401e-a88f-087ddcdd077c-tls-certs\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.190728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.190578    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-home\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.190728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.190614    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v584j\" (UniqueName: \"kubernetes.io/projected/5b34fc68-99bf-401e-a88f-087ddcdd077c-kube-api-access-v584j\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.190728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.190662    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-tmp-dir\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.190842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.190737    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-dshm\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.190842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.190755    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-model-cache\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.191138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.191107    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-home\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.191138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.191125    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-tmp-dir\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.191288 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.191221    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-model-cache\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.192909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.192867    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-dshm\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.193302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.193282    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34fc68-99bf-401e-a88f-087ddcdd077c-tls-certs\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.223504 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.223474    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v584j\" (UniqueName: \"kubernetes.io/projected/5b34fc68-99bf-401e-a88f-087ddcdd077c-kube-api-access-v584j\") pod \"premium-w3-92f6b3-kserve-776f8fff4c-gs2sv\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") " pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.370092 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.370004    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:36.671790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.671759    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"]
Oct 29 23:28:36.673011 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:28:36.672971    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b34fc68_99bf_401e_a88f_087ddcdd077c.slice/crio-4782271ff01287e1ba31543748aeaa75bad839fa292109349ae09f7698fdb4e2 WatchSource:0}: Error finding container 4782271ff01287e1ba31543748aeaa75bad839fa292109349ae09f7698fdb4e2: Status 404 returned error can't find the container with id 4782271ff01287e1ba31543748aeaa75bad839fa292109349ae09f7698fdb4e2
Oct 29 23:28:36.821629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.821597    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" event={"ID":"5b34fc68-99bf-401e-a88f-087ddcdd077c","Type":"ContainerStarted","Data":"e9ad17668be42c8753425e0ba4f865edf3cd1d766b8b67ea59b83e67e88d188d"}
Oct 29 23:28:36.821809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.821638    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" event={"ID":"5b34fc68-99bf-401e-a88f-087ddcdd077c","Type":"ContainerStarted","Data":"4782271ff01287e1ba31543748aeaa75bad839fa292109349ae09f7698fdb4e2"}
Oct 29 23:28:36.869552 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:36.869505    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podStartSLOduration=0.869492242 podStartE2EDuration="869.492242ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:36.868737008 +0000 UTC m=+894.972997395" watchObservedRunningTime="2026-10-29 23:28:36.869492242 +0000 UTC m=+894.973752656"
Oct 29 23:28:37.370931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:37.370872    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:39.113266 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.113229    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"]
Oct 29 23:28:39.119020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.118986    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.139684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.139653    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ae1364-openshift-default-dockercfg-9nkvr\""
Oct 29 23:28:39.156105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.156075    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"]
Oct 29 23:28:39.224635 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224598    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150fb858-8a31-46dc-852c-46af8df85600-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.224635 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224645    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.224857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224729    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.224857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224772    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8jg\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-kube-api-access-tx8jg\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.224857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224836    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.225013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224859    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.225013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224896    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150fb858-8a31-46dc-852c-46af8df85600-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.225013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.224919    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.225124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.225011    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.325954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.325918    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.325989    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326040    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150fb858-8a31-46dc-852c-46af8df85600-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326074    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326109    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326126    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8jg\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-kube-api-access-tx8jg\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326197    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326227    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326248    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150fb858-8a31-46dc-852c-46af8df85600-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326349    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326392    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326594    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326601    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.326941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.326922    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150fb858-8a31-46dc-852c-46af8df85600-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.328509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.328487    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.328771 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.328752    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150fb858-8a31-46dc-852c-46af8df85600-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.352317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.352282    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.352467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.352449    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8jg\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-kube-api-access-tx8jg\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.435742 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.435711    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:39.602629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.602592    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:39.615155 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.615119    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:39.767623 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.767596    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"]
Oct 29 23:28:39.768504 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:28:39.768475    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150fb858_8a31_46dc_852c_46af8df85600.slice/crio-cb7b22da5e546fcf13b784d29ff905152c316f3e866c705f985ca65d60b1fc0d WatchSource:0}: Error finding container cb7b22da5e546fcf13b784d29ff905152c316f3e866c705f985ca65d60b1fc0d: Status 404 returned error can't find the container with id cb7b22da5e546fcf13b784d29ff905152c316f3e866c705f985ca65d60b1fc0d
Oct 29 23:28:39.770950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.770917    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:28:39.771031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.770990    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:28:39.771031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.771018    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:28:39.838353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.838323    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" event={"ID":"150fb858-8a31-46dc-852c-46af8df85600","Type":"ContainerStarted","Data":"c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26"}
Oct 29 23:28:39.838463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.838389    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" event={"ID":"150fb858-8a31-46dc-852c-46af8df85600","Type":"ContainerStarted","Data":"cb7b22da5e546fcf13b784d29ff905152c316f3e866c705f985ca65d60b1fc0d"}
Oct 29 23:28:39.850300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.850267    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:28:39.898698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:39.898639    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podStartSLOduration=0.89862059 podStartE2EDuration="898.62059ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:28:39.898544174 +0000 UTC m=+898.002804562" watchObservedRunningTime="2026-10-29 23:28:39.89862059 +0000 UTC m=+898.002880976"
Oct 29 23:28:39.927503 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:28:39.927464    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:28:40.436020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:40.435981    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:40.437801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:40.437780    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:28:40.437890 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:40.437829    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 23:28:41.440260 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:41.440220    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:28:41.440862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:41.440829    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 23:28:42.439417 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:42.439378    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:28:42.439786 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:42.439743    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 23:28:43.449903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:43.448339    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:28:43.449903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:43.448406    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 23:28:44.448903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:44.448297    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:28:44.448903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:44.448362    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 23:28:45.452906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:45.447984    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:28:45.452906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:45.448060    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 23:28:46.370510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:46.370470    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:46.383601 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:46.383569    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:46.436364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:46.436325    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 29 23:28:46.436518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:46.436377    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 29 23:28:46.897982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:46.897950    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:28:47.440964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:47.440933    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:47.441250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:47.441226    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:28:47.441961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:28:47.441943    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:29:05.889372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.889299    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"]
Oct 29 23:29:05.893312 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.893291    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:05.911268 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.911240    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:29:05.911428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.911238    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:29:05.925726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.925698    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:29:05.925726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.925712    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:29:05.927839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.927814    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"]
Oct 29 23:29:05.959850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.959822    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:05.960010 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.959854    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:05.960010 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:05.959949    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ftg\" (UniqueName: \"kubernetes.io/projected/a0da7a58-0c4d-494a-89c6-f7be452f4095-kube-api-access-l2ftg\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.061416 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.061384    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ftg\" (UniqueName: \"kubernetes.io/projected/a0da7a58-0c4d-494a-89c6-f7be452f4095-kube-api-access-l2ftg\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.061848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.061816    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.061999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.061866    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.062557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.062531    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.064258 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.064241    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.087898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.087852    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ftg\" (UniqueName: \"kubernetes.io/projected/a0da7a58-0c4d-494a-89c6-f7be452f4095-kube-api-access-l2ftg\") pod \"maas-api-revoke-sub-172702af-fvzxf\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.208001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.207938    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:06.513854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.513825    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"]
Oct 29 23:29:06.515244 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:06.515214    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0da7a58_0c4d_494a_89c6_f7be452f4095.slice/crio-d51263a4b324a985edac0fdb242fe68580e4b8bc9fb94624421da23afb5f6daa WatchSource:0}: Error finding container d51263a4b324a985edac0fdb242fe68580e4b8bc9fb94624421da23afb5f6daa: Status 404 returned error can't find the container with id d51263a4b324a985edac0fdb242fe68580e4b8bc9fb94624421da23afb5f6daa
Oct 29 23:29:06.728755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.728717    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"]
Oct 29 23:29:06.733688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.733665    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.752618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.752594    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2a53c9\"/\"openshift-service-ca.crt\""
Oct 29 23:29:06.752737 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.752669    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2a53c9\"/\"kube-root-ca.crt\""
Oct 29 23:29:06.755457 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.755429    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"]
Oct 29 23:29:06.766759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.766693    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2a53c9\"/\"default-dockercfg-8zffd\""
Oct 29 23:29:06.766870 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.766791    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2a53c9\"/\"premium-w1-2a53c9-kserve-self-signed-certs\""
Oct 29 23:29:06.769377 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.769350    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-dshm\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.769463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.769391    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-tmp-dir\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.769523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.769493    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhwh\" (UniqueName: \"kubernetes.io/projected/c580d5bd-73c2-4010-8087-61d8100e8df7-kube-api-access-tmhwh\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.769660 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.769637    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580d5bd-73c2-4010-8087-61d8100e8df7-tls-certs\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.769680    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-model-cache\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.769826 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.769810    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-home\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.870535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.870465    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-home\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.870790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.870615    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-dshm\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.870790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.870659    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-tmp-dir\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.870790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.870705    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhwh\" (UniqueName: \"kubernetes.io/projected/c580d5bd-73c2-4010-8087-61d8100e8df7-kube-api-access-tmhwh\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.871000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.870797    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580d5bd-73c2-4010-8087-61d8100e8df7-tls-certs\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.871000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.870982    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-home\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.871214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.870826    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-model-cache\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.871351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.871331    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-tmp-dir\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.871423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.871387    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-model-cache\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.873098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.873078    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-dshm\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.873324 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.873307    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580d5bd-73c2-4010-8087-61d8100e8df7-tls-certs\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.896865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.896837    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhwh\" (UniqueName: \"kubernetes.io/projected/c580d5bd-73c2-4010-8087-61d8100e8df7-kube-api-access-tmhwh\") pod \"premium-w1-2a53c9-kserve-756f558846-8vrwc\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") " pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:06.998126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:06.998093    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf" event={"ID":"a0da7a58-0c4d-494a-89c6-f7be452f4095","Type":"ContainerStarted","Data":"d51263a4b324a985edac0fdb242fe68580e4b8bc9fb94624421da23afb5f6daa"}
Oct 29 23:29:07.047056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:07.046971    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:07.353465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:07.353432    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"]
Oct 29 23:29:07.354600 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:07.354573    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc580d5bd_73c2_4010_8087_61d8100e8df7.slice/crio-4e3c6817dc29880111b49a558725ccfec929c21ea25fead391e438fc0e393a37 WatchSource:0}: Error finding container 4e3c6817dc29880111b49a558725ccfec929c21ea25fead391e438fc0e393a37: Status 404 returned error can't find the container with id 4e3c6817dc29880111b49a558725ccfec929c21ea25fead391e438fc0e393a37
Oct 29 23:29:08.007654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:08.007612    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" event={"ID":"c580d5bd-73c2-4010-8087-61d8100e8df7","Type":"ContainerStarted","Data":"bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b"}
Oct 29 23:29:08.008194 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:08.007673    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" event={"ID":"c580d5bd-73c2-4010-8087-61d8100e8df7","Type":"ContainerStarted","Data":"4e3c6817dc29880111b49a558725ccfec929c21ea25fead391e438fc0e393a37"}
Oct 29 23:29:08.047529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:08.047486    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:09.924426 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:29:09.924391    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:29:10.022518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:10.022482    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf" event={"ID":"a0da7a58-0c4d-494a-89c6-f7be452f4095","Type":"ContainerStarted","Data":"909d3c5da8d40e7c34bc797063172c93c76723c9e133c6dfbbe00518f13af1e4"}
Oct 29 23:29:10.108218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:10.108157    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf" podStartSLOduration=1.978470115 podStartE2EDuration="5.108142985s" totalImagesPullingTime="3.12967287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:10.10479329 +0000 UTC m=+928.209053676" watchObservedRunningTime="2026-10-29 23:29:10.108142985 +0000 UTC m=+928.212403363"
Oct 29 23:29:10.108651 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:10.108626    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podStartSLOduration=4.108621498 podStartE2EDuration="4.108621498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:08.077346005 +0000 UTC m=+926.181606393" watchObservedRunningTime="2026-10-29 23:29:10.108621498 +0000 UTC m=+928.212881950"
Oct 29 23:29:11.028961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:11.028930    6203 generic.go:356] "Generic (PLEG): container finished" podID="a0da7a58-0c4d-494a-89c6-f7be452f4095" containerID="909d3c5da8d40e7c34bc797063172c93c76723c9e133c6dfbbe00518f13af1e4" exitCode=0
Oct 29 23:29:11.029366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:11.029019    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf" event={"ID":"a0da7a58-0c4d-494a-89c6-f7be452f4095","Type":"ContainerDied","Data":"909d3c5da8d40e7c34bc797063172c93c76723c9e133c6dfbbe00518f13af1e4"}
Oct 29 23:29:12.209018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.208990    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:12.283781 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.283743    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-xks-ca\") pod \"a0da7a58-0c4d-494a-89c6-f7be452f4095\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") "
Oct 29 23:29:12.284030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.283915    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-openshift-ca\") pod \"a0da7a58-0c4d-494a-89c6-f7be452f4095\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") "
Oct 29 23:29:12.284030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.284022    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0da7a58-0c4d-494a-89c6-f7be452f4095-kube-api-access-l2ftg\" (UniqueName: \"kubernetes.io/projected/a0da7a58-0c4d-494a-89c6-f7be452f4095-kube-api-access-l2ftg\") pod \"a0da7a58-0c4d-494a-89c6-f7be452f4095\" (UID: \"a0da7a58-0c4d-494a-89c6-f7be452f4095\") "
Oct 29 23:29:12.284421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.284370    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-openshift-ca" pod "a0da7a58-0c4d-494a-89c6-f7be452f4095" (UID: "a0da7a58-0c4d-494a-89c6-f7be452f4095"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:12.284553 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.284466    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:12.286215 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.286151    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-xks-ca" pod "a0da7a58-0c4d-494a-89c6-f7be452f4095" (UID: "a0da7a58-0c4d-494a-89c6-f7be452f4095"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:12.286779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.286758    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0da7a58-0c4d-494a-89c6-f7be452f4095-kube-api-access-l2ftg" pod "a0da7a58-0c4d-494a-89c6-f7be452f4095" (UID: "a0da7a58-0c4d-494a-89c6-f7be452f4095"). InnerVolumeSpecName "kube-api-access-l2ftg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:12.385224 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.385177    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0da7a58-0c4d-494a-89c6-f7be452f4095-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:12.385224 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.385213    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ftg\" (UniqueName: \"kubernetes.io/projected/a0da7a58-0c4d-494a-89c6-f7be452f4095-kube-api-access-l2ftg\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:12.935682 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.935638    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"]
Oct 29 23:29:12.939900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.936387    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a0da7a58-0c4d-494a-89c6-f7be452f4095" containerName="revoke-keys"
Oct 29 23:29:12.939900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.936409    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0da7a58-0c4d-494a-89c6-f7be452f4095" containerName="revoke-keys"
Oct 29 23:29:12.939900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.936562    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0da7a58-0c4d-494a-89c6-f7be452f4095" containerName="revoke-keys"
Oct 29 23:29:12.941301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.940607    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:12.959979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.959957    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ae1784\"/\"openshift-service-ca.crt\""
Oct 29 23:29:12.977512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.977488    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ae1784\"/\"default-dockercfg-95pjs\""
Oct 29 23:29:12.977647 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.977530    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ae1784\"/\"premium-w6-ae1784-kserve-self-signed-certs\""
Oct 29 23:29:12.978188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.978161    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"]
Oct 29 23:29:12.993596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.993570    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-model-cache\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:12.993712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.993620    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-home\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:12.993759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.993725    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrxz\" (UniqueName: \"kubernetes.io/projected/6d706de1-efaf-4c90-b710-eab26d3ed7f8-kube-api-access-bcrxz\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:12.993803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.993765    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-dshm\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:12.993969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.993950    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tls-certs\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:12.994046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:12.994002    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tmp-dir\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.041269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.041232    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf" event={"ID":"a0da7a58-0c4d-494a-89c6-f7be452f4095","Type":"ContainerDied","Data":"d51263a4b324a985edac0fdb242fe68580e4b8bc9fb94624421da23afb5f6daa"}
Oct 29 23:29:13.041499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.041311    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d51263a4b324a985edac0fdb242fe68580e4b8bc9fb94624421da23afb5f6daa"
Oct 29 23:29:13.041499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.041339    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"
Oct 29 23:29:13.060666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.060641    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ae1784\"/\"kube-root-ca.crt\""
Oct 29 23:29:13.094952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.094912    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tmp-dir\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095135 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095029    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-model-cache\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095135 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095067    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-home\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095164    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrxz\" (UniqueName: \"kubernetes.io/projected/6d706de1-efaf-4c90-b710-eab26d3ed7f8-kube-api-access-bcrxz\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095252    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-dshm\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095388 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095358    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tmp-dir\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095388 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095375    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-model-cache\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095471    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tls-certs\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.095629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.095604    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-home\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.097905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.097854    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tls-certs\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.098095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.098073    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-dshm\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.129706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.124214    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrxz\" (UniqueName: \"kubernetes.io/projected/6d706de1-efaf-4c90-b710-eab26d3ed7f8-kube-api-access-bcrxz\") pod \"premium-w6-ae1784-kserve-5584cc646-fnr8h\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") " pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.261909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.261795    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:13.569488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:13.569442    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"]
Oct 29 23:29:13.572172 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:13.572129    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d706de1_efaf_4c90_b710_eab26d3ed7f8.slice/crio-c014f22b482d0dd826e3597d19fdf9042f6a6911500c24235398a7e1ef81318f WatchSource:0}: Error finding container c014f22b482d0dd826e3597d19fdf9042f6a6911500c24235398a7e1ef81318f: Status 404 returned error can't find the container with id c014f22b482d0dd826e3597d19fdf9042f6a6911500c24235398a7e1ef81318f
Oct 29 23:29:14.047788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:14.047749    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" event={"ID":"6d706de1-efaf-4c90-b710-eab26d3ed7f8","Type":"ContainerStarted","Data":"568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc"}
Oct 29 23:29:14.047788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:14.047793    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" event={"ID":"6d706de1-efaf-4c90-b710-eab26d3ed7f8","Type":"ContainerStarted","Data":"c014f22b482d0dd826e3597d19fdf9042f6a6911500c24235398a7e1ef81318f"}
Oct 29 23:29:14.096528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:14.096478    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podStartSLOduration=2.096465544 podStartE2EDuration="2.096465544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:14.095776508 +0000 UTC m=+932.200036893" watchObservedRunningTime="2026-10-29 23:29:14.096465544 +0000 UTC m=+932.200725931"
Oct 29 23:29:14.262420 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:14.262389    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:17.047695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:17.047653    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:17.060602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:17.060572    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:17.080897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:17.080855    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:29:19.591592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.591542    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"]
Oct 29 23:29:19.596614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.596579    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.629152 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.629107    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:29:19.629334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.629181    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:29:19.629334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.629251    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:29:19.629334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.629277    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:29:19.633711 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.633678    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"]
Oct 29 23:29:19.686931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.684597    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.686931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.684666    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.686931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.684820    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crq54\" (UniqueName: \"kubernetes.io/projected/89242841-51c2-4bc9-977f-ed1020eb21b9-kube-api-access-crq54\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.785845 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.785792    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crq54\" (UniqueName: \"kubernetes.io/projected/89242841-51c2-4bc9-977f-ed1020eb21b9-kube-api-access-crq54\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.786101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.785922    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.786101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.785970    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.788242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.788208    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.791521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.791375    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.815617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.815569    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crq54\" (UniqueName: \"kubernetes.io/projected/89242841-51c2-4bc9-977f-ed1020eb21b9-kube-api-access-crq54\") pod \"maas-api-revoke-sub-8d9ea233-blmg5\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:19.914757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:19.914663    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:20.325548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:20.324432    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"]
Oct 29 23:29:21.007005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.006966    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"]
Oct 29 23:29:21.011942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.011920    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.032789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.032759    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"]
Oct 29 23:29:21.105809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.105777    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5" event={"ID":"89242841-51c2-4bc9-977f-ed1020eb21b9","Type":"ContainerStarted","Data":"9630a040c2492228b0ea7a9609742fb4a717b9f98ddbde8085e2b352a6074e4e"}
Oct 29 23:29:21.106023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.105824    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5" event={"ID":"89242841-51c2-4bc9-977f-ed1020eb21b9","Type":"ContainerStarted","Data":"98794b04ea098f5b1d9728e367a1967cef9574cbefd8b57d570b45f1cb842f21"}
Oct 29 23:29:21.111508 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.111479    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t5sz\" (UniqueName: \"kubernetes.io/projected/cbb18175-ae67-4e52-a022-00368741f817-kube-api-access-7t5sz\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.111749 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.111729    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb18175-ae67-4e52-a022-00368741f817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.111829 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.111787    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb18175-ae67-4e52-a022-00368741f817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.155013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.154956    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5" podStartSLOduration=2.1549411 podStartE2EDuration="2.1549411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:21.152915788 +0000 UTC m=+939.257176169" watchObservedRunningTime="2026-10-29 23:29:21.1549411 +0000 UTC m=+939.259201485"
Oct 29 23:29:21.213106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.213064    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb18175-ae67-4e52-a022-00368741f817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.213284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.213130    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb18175-ae67-4e52-a022-00368741f817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.213284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.213202    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t5sz\" (UniqueName: \"kubernetes.io/projected/cbb18175-ae67-4e52-a022-00368741f817-kube-api-access-7t5sz\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.213866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.213808    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb18175-ae67-4e52-a022-00368741f817-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.215908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.215849    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb18175-ae67-4e52-a022-00368741f817-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.239787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.239756    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t5sz\" (UniqueName: \"kubernetes.io/projected/cbb18175-ae67-4e52-a022-00368741f817-kube-api-access-7t5sz\") pod \"maas-api-revoke-sub-b3049dd0-r849b\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.327359 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.327268    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:21.642670 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:21.642640    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb18175_ae67_4e52_a022_00368741f817.slice/crio-04389db1ded3a436999fd0149085aaf7c2b7d850edf95127901217289a3efc4b WatchSource:0}: Error finding container 04389db1ded3a436999fd0149085aaf7c2b7d850edf95127901217289a3efc4b: Status 404 returned error can't find the container with id 04389db1ded3a436999fd0149085aaf7c2b7d850edf95127901217289a3efc4b
Oct 29 23:29:21.643070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:21.643046    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"]
Oct 29 23:29:22.113186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:22.113147    6203 generic.go:356] "Generic (PLEG): container finished" podID="89242841-51c2-4bc9-977f-ed1020eb21b9" containerID="9630a040c2492228b0ea7a9609742fb4a717b9f98ddbde8085e2b352a6074e4e" exitCode=0
Oct 29 23:29:22.113633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:22.113226    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5" event={"ID":"89242841-51c2-4bc9-977f-ed1020eb21b9","Type":"ContainerDied","Data":"9630a040c2492228b0ea7a9609742fb4a717b9f98ddbde8085e2b352a6074e4e"}
Oct 29 23:29:22.114795 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:22.114774    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b" event={"ID":"cbb18175-ae67-4e52-a022-00368741f817","Type":"ContainerStarted","Data":"9a759885cb698474ac63ef1379da39eb80fd357794d3f31f40dbbeabe9a0db3d"}
Oct 29 23:29:22.114899 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:22.114811    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b" event={"ID":"cbb18175-ae67-4e52-a022-00368741f817","Type":"ContainerStarted","Data":"04389db1ded3a436999fd0149085aaf7c2b7d850edf95127901217289a3efc4b"}
Oct 29 23:29:22.207398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:22.207283    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b" podStartSLOduration=2.207264599 podStartE2EDuration="2.207264599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:22.204780569 +0000 UTC m=+940.309040956" watchObservedRunningTime="2026-10-29 23:29:22.207264599 +0000 UTC m=+940.311524985"
Oct 29 23:29:23.122516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.122481    6203 generic.go:356] "Generic (PLEG): container finished" podID="cbb18175-ae67-4e52-a022-00368741f817" containerID="9a759885cb698474ac63ef1379da39eb80fd357794d3f31f40dbbeabe9a0db3d" exitCode=0
Oct 29 23:29:23.122964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.122570    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b" event={"ID":"cbb18175-ae67-4e52-a022-00368741f817","Type":"ContainerDied","Data":"9a759885cb698474ac63ef1379da39eb80fd357794d3f31f40dbbeabe9a0db3d"}
Oct 29 23:29:23.262512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.262480    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:23.275760 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.275734    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:23.280428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.280401    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:23.367388 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.367355    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"]
Oct 29 23:29:23.367973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.367950    6203 cpu_manager.go:414] "containerMap: removing container" podUID="89242841-51c2-4bc9-977f-ed1020eb21b9" containerName="revoke-keys"
Oct 29 23:29:23.367973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.367973    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89242841-51c2-4bc9-977f-ed1020eb21b9" containerName="revoke-keys"
Oct 29 23:29:23.368137 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.368125    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="89242841-51c2-4bc9-977f-ed1020eb21b9" containerName="revoke-keys"
Oct 29 23:29:23.371687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.371663    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.395203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.395174    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-92f6b3-openshift-default-dockercfg-lbv8z\""
Oct 29 23:29:23.404118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.404083    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"]
Oct 29 23:29:23.436265 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.436236    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-openshift-ca\") pod \"89242841-51c2-4bc9-977f-ed1020eb21b9\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") "
Oct 29 23:29:23.436413 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.436273    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-xks-ca\") pod \"89242841-51c2-4bc9-977f-ed1020eb21b9\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") "
Oct 29 23:29:23.436455 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.436421    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89242841-51c2-4bc9-977f-ed1020eb21b9-kube-api-access-crq54\" (UniqueName: \"kubernetes.io/projected/89242841-51c2-4bc9-977f-ed1020eb21b9-kube-api-access-crq54\") pod \"89242841-51c2-4bc9-977f-ed1020eb21b9\" (UID: \"89242841-51c2-4bc9-977f-ed1020eb21b9\") "
Oct 29 23:29:23.436685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.436659    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-openshift-ca" pod "89242841-51c2-4bc9-977f-ed1020eb21b9" (UID: "89242841-51c2-4bc9-977f-ed1020eb21b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:23.436781 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.436756    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.436908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.436854    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437045 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.436980    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.437040    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.437093    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.437122    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbnt\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-kube-api-access-fpbnt\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.437185    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd20e7cd-7216-4eb4-8880-6f882e08aead-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.437204    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.437222    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.437384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.437302    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:23.438586 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.438561    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89242841-51c2-4bc9-977f-ed1020eb21b9-kube-api-access-crq54" pod "89242841-51c2-4bc9-977f-ed1020eb21b9" (UID: "89242841-51c2-4bc9-977f-ed1020eb21b9"). InnerVolumeSpecName "kube-api-access-crq54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:23.438696 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.438626    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-xks-ca" pod "89242841-51c2-4bc9-977f-ed1020eb21b9" (UID: "89242841-51c2-4bc9-977f-ed1020eb21b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:23.538541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538507    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538549    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538574    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbnt\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-kube-api-access-fpbnt\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538606    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd20e7cd-7216-4eb4-8880-6f882e08aead-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538622    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538640    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538722    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538761    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538832    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.538949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538942    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89242841-51c2-4bc9-977f-ed1020eb21b9-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:23.539180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.538961    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crq54\" (UniqueName: \"kubernetes.io/projected/89242841-51c2-4bc9-977f-ed1020eb21b9-kube-api-access-crq54\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:23.539180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.539038    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.539180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.539167    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.539417 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.539394    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.539534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.539479    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.539578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.539553    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd20e7cd-7216-4eb4-8880-6f882e08aead-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.540937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.540919    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.541186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.541167    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.567064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.567040    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbnt\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-kube-api-access-fpbnt\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.567512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.567493    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:23.694926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:23.694871    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:24.044347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.044316    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"]
Oct 29 23:29:24.045264 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:24.045242    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd20e7cd_7216_4eb4_8880_6f882e08aead.slice/crio-470de3ed7dae3a05bf0f6d779f889c3e2bdc6d0f7e25c9d87968b041de63b27c WatchSource:0}: Error finding container 470de3ed7dae3a05bf0f6d779f889c3e2bdc6d0f7e25c9d87968b041de63b27c: Status 404 returned error can't find the container with id 470de3ed7dae3a05bf0f6d779f889c3e2bdc6d0f7e25c9d87968b041de63b27c
Oct 29 23:29:24.048275 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.048238    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:24.048386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.048314    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:24.048386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.048359    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:24.131028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.131001    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"
Oct 29 23:29:24.131400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.131039    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5" event={"ID":"89242841-51c2-4bc9-977f-ed1020eb21b9","Type":"ContainerDied","Data":"98794b04ea098f5b1d9728e367a1967cef9574cbefd8b57d570b45f1cb842f21"}
Oct 29 23:29:24.131400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.131088    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98794b04ea098f5b1d9728e367a1967cef9574cbefd8b57d570b45f1cb842f21"
Oct 29 23:29:24.133104 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.133018    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" event={"ID":"cd20e7cd-7216-4eb4-8880-6f882e08aead","Type":"ContainerStarted","Data":"757122e7823460841fca55431016fde9e927cb96416b57b858f94ecb6b65aa13"}
Oct 29 23:29:24.133104 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.133066    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" event={"ID":"cd20e7cd-7216-4eb4-8880-6f882e08aead","Type":"ContainerStarted","Data":"470de3ed7dae3a05bf0f6d779f889c3e2bdc6d0f7e25c9d87968b041de63b27c"}
Oct 29 23:29:24.147894 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.147824    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:29:24.223160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.222369    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podStartSLOduration=1.22234729 podStartE2EDuration="1.22234729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:24.222301021 +0000 UTC m=+942.326561408" watchObservedRunningTime="2026-10-29 23:29:24.22234729 +0000 UTC m=+942.326607678"
Oct 29 23:29:24.290403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.290378    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:24.349222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.348366    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb18175-ae67-4e52-a022-00368741f817-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb18175-ae67-4e52-a022-00368741f817-maas-api-xks-ca\") pod \"cbb18175-ae67-4e52-a022-00368741f817\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") "
Oct 29 23:29:24.349222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.348507    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb18175-ae67-4e52-a022-00368741f817-kube-api-access-7t5sz\" (UniqueName: \"kubernetes.io/projected/cbb18175-ae67-4e52-a022-00368741f817-kube-api-access-7t5sz\") pod \"cbb18175-ae67-4e52-a022-00368741f817\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") "
Oct 29 23:29:24.349222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.348563    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb18175-ae67-4e52-a022-00368741f817-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb18175-ae67-4e52-a022-00368741f817-maas-api-openshift-ca\") pod \"cbb18175-ae67-4e52-a022-00368741f817\" (UID: \"cbb18175-ae67-4e52-a022-00368741f817\") "
Oct 29 23:29:24.349706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.349663    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb18175-ae67-4e52-a022-00368741f817-maas-api-openshift-ca" pod "cbb18175-ae67-4e52-a022-00368741f817" (UID: "cbb18175-ae67-4e52-a022-00368741f817"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:24.359328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.357297    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb18175-ae67-4e52-a022-00368741f817-maas-api-xks-ca" pod "cbb18175-ae67-4e52-a022-00368741f817" (UID: "cbb18175-ae67-4e52-a022-00368741f817"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:24.359328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.359276    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb18175-ae67-4e52-a022-00368741f817-kube-api-access-7t5sz" pod "cbb18175-ae67-4e52-a022-00368741f817" (UID: "cbb18175-ae67-4e52-a022-00368741f817"). InnerVolumeSpecName "kube-api-access-7t5sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:24.451901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.451701    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbb18175-ae67-4e52-a022-00368741f817-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:24.451901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.451736    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t5sz\" (UniqueName: \"kubernetes.io/projected/cbb18175-ae67-4e52-a022-00368741f817-kube-api-access-7t5sz\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:24.451901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.451750    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbb18175-ae67-4e52-a022-00368741f817-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:24.695123 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:24.695090    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:25.159301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:25.159218    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"
Oct 29 23:29:25.159301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:25.159218    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b" event={"ID":"cbb18175-ae67-4e52-a022-00368741f817","Type":"ContainerDied","Data":"04389db1ded3a436999fd0149085aaf7c2b7d850edf95127901217289a3efc4b"}
Oct 29 23:29:25.159301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:25.159289    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04389db1ded3a436999fd0149085aaf7c2b7d850edf95127901217289a3efc4b"
Oct 29 23:29:25.696701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:25.696665    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 23:29:25.696863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:25.696733    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 23:29:26.696107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:26.696058    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 23:29:26.696488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:26.696125    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 23:29:27.695267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:27.695231    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 23:29:27.695451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:27.695283    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 23:29:28.696160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:28.696122    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 23:29:28.696631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:28.696187    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 23:29:29.696029 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:29.695992    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 23:29:29.696188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:29.696048    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 23:29:30.791208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:30.791174    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:30.791890 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:30.791849    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:30.792829 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:30.792800    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:29:35.404576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.404544    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"]
Oct 29 23:29:35.405309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.405287    6203 cpu_manager.go:414] "containerMap: removing container" podUID="cbb18175-ae67-4e52-a022-00368741f817" containerName="revoke-keys"
Oct 29 23:29:35.405357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.405313    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb18175-ae67-4e52-a022-00368741f817" containerName="revoke-keys"
Oct 29 23:29:35.405479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.405468    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb18175-ae67-4e52-a022-00368741f817" containerName="revoke-keys"
Oct 29 23:29:35.409165 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.409144    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.428104 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.428078    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"]
Oct 29 23:29:35.440203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.440173    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:29:35.440334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.440308    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:29:35.440395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.440360    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:29:35.440668 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.440654    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:29:35.491616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.491578    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.491616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.491618    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgxg\" (UniqueName: \"kubernetes.io/projected/7ec21f45-e3a9-4aff-b528-d5babb961453-kube-api-access-lkgxg\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.491853 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.491805    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.593484 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.593443    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.593484 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.593486    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgxg\" (UniqueName: \"kubernetes.io/projected/7ec21f45-e3a9-4aff-b528-d5babb961453-kube-api-access-lkgxg\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.593700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.593612    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.594177 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.594160    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.596068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.596046    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.618983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.618955    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgxg\" (UniqueName: \"kubernetes.io/projected/7ec21f45-e3a9-4aff-b528-d5babb961453-kube-api-access-lkgxg\") pod \"maas-api-revoke-sub-4491bb74-4w4r4\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:35.722854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:35.722819    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:36.032127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:36.032099    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"]
Oct 29 23:29:36.037008 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:36.036981    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec21f45_e3a9_4aff_b528_d5babb961453.slice/crio-96a2e53e3949a0d0e783dee0d3963d8c76eddd50cba10aaa6ce3bb3cd0b98faa WatchSource:0}: Error finding container 96a2e53e3949a0d0e783dee0d3963d8c76eddd50cba10aaa6ce3bb3cd0b98faa: Status 404 returned error can't find the container with id 96a2e53e3949a0d0e783dee0d3963d8c76eddd50cba10aaa6ce3bb3cd0b98faa
Oct 29 23:29:36.227187 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:36.227145    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4" event={"ID":"7ec21f45-e3a9-4aff-b528-d5babb961453","Type":"ContainerStarted","Data":"96a2e53e3949a0d0e783dee0d3963d8c76eddd50cba10aaa6ce3bb3cd0b98faa"}
Oct 29 23:29:37.214987 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.214912    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"]
Oct 29 23:29:37.218805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.218789    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.233979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.233955    6203 generic.go:356] "Generic (PLEG): container finished" podID="7ec21f45-e3a9-4aff-b528-d5babb961453" containerID="6326991daf5f2b83cd412606f6d7cf1ca18f092f0c898645406245c7d3b0fb3d" exitCode=0
Oct 29 23:29:37.234079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.234024    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4" event={"ID":"7ec21f45-e3a9-4aff-b528-d5babb961453","Type":"ContainerDied","Data":"6326991daf5f2b83cd412606f6d7cf1ca18f092f0c898645406245c7d3b0fb3d"}
Oct 29 23:29:37.240490 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.240461    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"]
Oct 29 23:29:37.324854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.324817    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.325035 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.325006    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8kj\" (UniqueName: \"kubernetes.io/projected/ed424d6f-3464-4853-b1f2-ec7eef095c5b-kube-api-access-gd8kj\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.325077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.325068    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.426225 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.426185    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8kj\" (UniqueName: \"kubernetes.io/projected/ed424d6f-3464-4853-b1f2-ec7eef095c5b-kube-api-access-gd8kj\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.426429 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.426274    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.426429 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.426340    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.427063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.427037    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.428583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.428556    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.452746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.452716    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8kj\" (UniqueName: \"kubernetes.io/projected/ed424d6f-3464-4853-b1f2-ec7eef095c5b-kube-api-access-gd8kj\") pod \"maas-api-revoke-sub-c021d202-lzz7x\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.532150 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.532044    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:37.835684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:37.835655    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"]
Oct 29 23:29:37.836659 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:37.836630    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded424d6f_3464_4853_b1f2_ec7eef095c5b.slice/crio-b686eb12f9cd1442795f7b39a15247047474a27583ab090a2a09b13ae51c867c WatchSource:0}: Error finding container b686eb12f9cd1442795f7b39a15247047474a27583ab090a2a09b13ae51c867c: Status 404 returned error can't find the container with id b686eb12f9cd1442795f7b39a15247047474a27583ab090a2a09b13ae51c867c
Oct 29 23:29:38.242369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.242331    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x" event={"ID":"ed424d6f-3464-4853-b1f2-ec7eef095c5b","Type":"ContainerStarted","Data":"f7acdfd490c92430d58f57afaa94322858d17c0a2c7900d682da19586583ba6a"}
Oct 29 23:29:38.242866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.242388    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x" event={"ID":"ed424d6f-3464-4853-b1f2-ec7eef095c5b","Type":"ContainerStarted","Data":"b686eb12f9cd1442795f7b39a15247047474a27583ab090a2a09b13ae51c867c"}
Oct 29 23:29:38.288518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.288453    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x" podStartSLOduration=1.288437098 podStartE2EDuration="1.288437098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:38.287306223 +0000 UTC m=+956.391566609" watchObservedRunningTime="2026-10-29 23:29:38.288437098 +0000 UTC m=+956.392697515"
Oct 29 23:29:38.413659 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.413628    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:38.547901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.547316    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-xks-ca\") pod \"7ec21f45-e3a9-4aff-b528-d5babb961453\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") "
Oct 29 23:29:38.547901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.547419    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec21f45-e3a9-4aff-b528-d5babb961453-kube-api-access-lkgxg\" (UniqueName: \"kubernetes.io/projected/7ec21f45-e3a9-4aff-b528-d5babb961453-kube-api-access-lkgxg\") pod \"7ec21f45-e3a9-4aff-b528-d5babb961453\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") "
Oct 29 23:29:38.547901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.547488    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-openshift-ca\") pod \"7ec21f45-e3a9-4aff-b528-d5babb961453\" (UID: \"7ec21f45-e3a9-4aff-b528-d5babb961453\") "
Oct 29 23:29:38.550927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.550747    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-openshift-ca" pod "7ec21f45-e3a9-4aff-b528-d5babb961453" (UID: "7ec21f45-e3a9-4aff-b528-d5babb961453"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:38.551209 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.551182    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:38.552993 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.552960    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-xks-ca" pod "7ec21f45-e3a9-4aff-b528-d5babb961453" (UID: "7ec21f45-e3a9-4aff-b528-d5babb961453"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:38.553635 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.553609    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec21f45-e3a9-4aff-b528-d5babb961453-kube-api-access-lkgxg" pod "7ec21f45-e3a9-4aff-b528-d5babb961453" (UID: "7ec21f45-e3a9-4aff-b528-d5babb961453"). InnerVolumeSpecName "kube-api-access-lkgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:38.652958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.652860    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec21f45-e3a9-4aff-b528-d5babb961453-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:38.652958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.652913    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgxg\" (UniqueName: \"kubernetes.io/projected/7ec21f45-e3a9-4aff-b528-d5babb961453-kube-api-access-lkgxg\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:38.838612 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.838572    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"]
Oct 29 23:29:38.839428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.839398    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7ec21f45-e3a9-4aff-b528-d5babb961453" containerName="revoke-keys"
Oct 29 23:29:38.839428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.839422    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec21f45-e3a9-4aff-b528-d5babb961453" containerName="revoke-keys"
Oct 29 23:29:38.839655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.839591    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec21f45-e3a9-4aff-b528-d5babb961453" containerName="revoke-keys"
Oct 29 23:29:38.842994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.842972    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.860532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.860503    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ae1784-openshift-default-dockercfg-bf8gx\""
Oct 29 23:29:38.878585 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.878553    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"]
Oct 29 23:29:38.940544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.940511    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"]
Oct 29 23:29:38.944386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.944370    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:38.955634 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.955602    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.955740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.955641    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.955740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.955658    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.955843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.955830    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.955893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.955868    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.955948 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.955925    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.955985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.955961    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.956032 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.956020    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjx76\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-kube-api-access-mjx76\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.956092 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.956080    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:38.964845 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:38.964817    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"]
Oct 29 23:29:39.057001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.056968    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.057166 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057010    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057166 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057031    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057166 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057084    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057208    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.057283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057233    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057278    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057350    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057393    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057454    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjx76\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-kube-api-access-mjx76\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057494    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057518    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfng\" (UniqueName: \"kubernetes.io/projected/cf87280a-7926-4446-85f5-294bf2a519ef-kube-api-access-tvfng\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.057534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057520    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057645    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.057812 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.057785    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.058025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.058005    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.058110 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.058095    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.059555 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.059535    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.059643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.059585    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.083954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.083922    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.084134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.084106    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjx76\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-kube-api-access-mjx76\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.157000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.156963    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:39.158437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.158414    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfng\" (UniqueName: \"kubernetes.io/projected/cf87280a-7926-4446-85f5-294bf2a519ef-kube-api-access-tvfng\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.158508 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.158462    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.158560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.158549    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.159255 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.159234    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.161016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.160991    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.184503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.184475    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfng\" (UniqueName: \"kubernetes.io/projected/cf87280a-7926-4446-85f5-294bf2a519ef-kube-api-access-tvfng\") pod \"maas-api-revoke-sub-8a7199a7-5rvmh\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.250856 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.250828    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"
Oct 29 23:29:39.251292 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.250864    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4" event={"ID":"7ec21f45-e3a9-4aff-b528-d5babb961453","Type":"ContainerDied","Data":"96a2e53e3949a0d0e783dee0d3963d8c76eddd50cba10aaa6ce3bb3cd0b98faa"}
Oct 29 23:29:39.251292 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.250962    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a2e53e3949a0d0e783dee0d3963d8c76eddd50cba10aaa6ce3bb3cd0b98faa"
Oct 29 23:29:39.256978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.256949    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:39.258729 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.258695    6203 generic.go:356] "Generic (PLEG): container finished" podID="ed424d6f-3464-4853-b1f2-ec7eef095c5b" containerID="f7acdfd490c92430d58f57afaa94322858d17c0a2c7900d682da19586583ba6a" exitCode=0
Oct 29 23:29:39.258852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.258745    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x" event={"ID":"ed424d6f-3464-4853-b1f2-ec7eef095c5b","Type":"ContainerDied","Data":"f7acdfd490c92430d58f57afaa94322858d17c0a2c7900d682da19586583ba6a"}
Oct 29 23:29:39.480611 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.480581    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"]
Oct 29 23:29:39.482328 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:39.482292    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd351e563_7bcd_4a78_b4f2_3e2ae3fcad10.slice/crio-b2eeca9a80d7f955be0107717026a608e69493989cb2f4a9140214ee3fda6e92 WatchSource:0}: Error finding container b2eeca9a80d7f955be0107717026a608e69493989cb2f4a9140214ee3fda6e92: Status 404 returned error can't find the container with id b2eeca9a80d7f955be0107717026a608e69493989cb2f4a9140214ee3fda6e92
Oct 29 23:29:39.485004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.484966    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:39.485093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.485036    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:39.485093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.485068    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:39.573320 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:39.573291    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf87280a_7926_4446_85f5_294bf2a519ef.slice/crio-325c5713497107d1cb28fd4ab83046025e32810d9efa54227649dec3c417ad47 WatchSource:0}: Error finding container 325c5713497107d1cb28fd4ab83046025e32810d9efa54227649dec3c417ad47: Status 404 returned error can't find the container with id 325c5713497107d1cb28fd4ab83046025e32810d9efa54227649dec3c417ad47
Oct 29 23:29:39.575121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:39.575094    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"]
Oct 29 23:29:39.928178 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:29:39.928111    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:29:40.266470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.266432    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" event={"ID":"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10","Type":"ContainerStarted","Data":"9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4"}
Oct 29 23:29:40.266955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.266503    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" event={"ID":"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10","Type":"ContainerStarted","Data":"b2eeca9a80d7f955be0107717026a608e69493989cb2f4a9140214ee3fda6e92"}
Oct 29 23:29:40.268307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.268278    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh" event={"ID":"cf87280a-7926-4446-85f5-294bf2a519ef","Type":"ContainerStarted","Data":"dcacb44081dc9ad10bc523063cff32c72b0bc5b14a8a32bab3c961afa303435c"}
Oct 29 23:29:40.268409 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.268327    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh" event={"ID":"cf87280a-7926-4446-85f5-294bf2a519ef","Type":"ContainerStarted","Data":"325c5713497107d1cb28fd4ab83046025e32810d9efa54227649dec3c417ad47"}
Oct 29 23:29:40.319023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.318973    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podStartSLOduration=2.318961271 podStartE2EDuration="2.318961271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:40.318154507 +0000 UTC m=+958.422414893" watchObservedRunningTime="2026-10-29 23:29:40.318961271 +0000 UTC m=+958.423221657"
Oct 29 23:29:40.452944 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.451263    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:40.472946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.472917    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-xks-ca\") pod \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") "
Oct 29 23:29:40.473082 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.472964    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed424d6f-3464-4853-b1f2-ec7eef095c5b-kube-api-access-gd8kj\" (UniqueName: \"kubernetes.io/projected/ed424d6f-3464-4853-b1f2-ec7eef095c5b-kube-api-access-gd8kj\") pod \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") "
Oct 29 23:29:40.473082 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.473077    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-openshift-ca\") pod \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\" (UID: \"ed424d6f-3464-4853-b1f2-ec7eef095c5b\") "
Oct 29 23:29:40.473505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.473429    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-openshift-ca" pod "ed424d6f-3464-4853-b1f2-ec7eef095c5b" (UID: "ed424d6f-3464-4853-b1f2-ec7eef095c5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:40.473567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.473550    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:40.475297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.475265    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-xks-ca" pod "ed424d6f-3464-4853-b1f2-ec7eef095c5b" (UID: "ed424d6f-3464-4853-b1f2-ec7eef095c5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:40.475297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.475284    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed424d6f-3464-4853-b1f2-ec7eef095c5b-kube-api-access-gd8kj" pod "ed424d6f-3464-4853-b1f2-ec7eef095c5b" (UID: "ed424d6f-3464-4853-b1f2-ec7eef095c5b"). InnerVolumeSpecName "kube-api-access-gd8kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:40.499820 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.499774    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh" podStartSLOduration=2.499757853 podStartE2EDuration="2.499757853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:40.369483381 +0000 UTC m=+958.473743770" watchObservedRunningTime="2026-10-29 23:29:40.499757853 +0000 UTC m=+958.604018279"
Oct 29 23:29:40.580701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.578408    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed424d6f-3464-4853-b1f2-ec7eef095c5b-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:40.580701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:40.578451    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd8kj\" (UniqueName: \"kubernetes.io/projected/ed424d6f-3464-4853-b1f2-ec7eef095c5b-kube-api-access-gd8kj\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:41.157201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.157153    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:41.159264 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.159228    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 23:29:41.159441 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.159290    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 23:29:41.162720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.162696    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"]
Oct 29 23:29:41.163260 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.163246    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ed424d6f-3464-4853-b1f2-ec7eef095c5b" containerName="revoke-keys"
Oct 29 23:29:41.163331 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.163261    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed424d6f-3464-4853-b1f2-ec7eef095c5b" containerName="revoke-keys"
Oct 29 23:29:41.163382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.163362    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed424d6f-3464-4853-b1f2-ec7eef095c5b" containerName="revoke-keys"
Oct 29 23:29:41.166838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.166818    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.185281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.185250    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.185479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.185453    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvn2\" (UniqueName: \"kubernetes.io/projected/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-kube-api-access-sjvn2\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.185643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.185625    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.189077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.189051    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"]
Oct 29 23:29:41.275887 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.275857    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"
Oct 29 23:29:41.276364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.275896    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x" event={"ID":"ed424d6f-3464-4853-b1f2-ec7eef095c5b","Type":"ContainerDied","Data":"b686eb12f9cd1442795f7b39a15247047474a27583ab090a2a09b13ae51c867c"}
Oct 29 23:29:41.276364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.275944    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b686eb12f9cd1442795f7b39a15247047474a27583ab090a2a09b13ae51c867c"
Oct 29 23:29:41.277661 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.277634    6203 generic.go:356] "Generic (PLEG): container finished" podID="cf87280a-7926-4446-85f5-294bf2a519ef" containerID="dcacb44081dc9ad10bc523063cff32c72b0bc5b14a8a32bab3c961afa303435c" exitCode=0
Oct 29 23:29:41.277804 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.277718    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh" event={"ID":"cf87280a-7926-4446-85f5-294bf2a519ef","Type":"ContainerDied","Data":"dcacb44081dc9ad10bc523063cff32c72b0bc5b14a8a32bab3c961afa303435c"}
Oct 29 23:29:41.287306 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.287282    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.287420 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.287311    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvn2\" (UniqueName: \"kubernetes.io/projected/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-kube-api-access-sjvn2\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.287420 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.287363    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.288081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.288051    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.289663 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.289645    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.317489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.317450    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvn2\" (UniqueName: \"kubernetes.io/projected/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-kube-api-access-sjvn2\") pod \"maas-api-revoke-sub-8a03f81e-r6f9x\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.481128 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.481090    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:41.814821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:41.814792    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"]
Oct 29 23:29:41.815547 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:41.815514    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9c42cf_9e27_4021_8682_3ca0e5d916f4.slice/crio-4fe3ed72bbee0d84a35585b0fab7e481c8978dd7a14f2ebd4eb19608ad24f95a WatchSource:0}: Error finding container 4fe3ed72bbee0d84a35585b0fab7e481c8978dd7a14f2ebd4eb19608ad24f95a: Status 404 returned error can't find the container with id 4fe3ed72bbee0d84a35585b0fab7e481c8978dd7a14f2ebd4eb19608ad24f95a
Oct 29 23:29:42.158757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.158669    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 23:29:42.158757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.158730    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 23:29:42.285478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.285436    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x" event={"ID":"3f9c42cf-9e27-4021-8682-3ca0e5d916f4","Type":"ContainerStarted","Data":"084e0be52fe399878629458d4fd18366e94a3f8ca7c1ca2cf2eceaa67ae7d19a"}
Oct 29 23:29:42.285925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.285534    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x" event={"ID":"3f9c42cf-9e27-4021-8682-3ca0e5d916f4","Type":"ContainerStarted","Data":"4fe3ed72bbee0d84a35585b0fab7e481c8978dd7a14f2ebd4eb19608ad24f95a"}
Oct 29 23:29:42.486482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.486449    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:42.500081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.500033    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf87280a-7926-4446-85f5-294bf2a519ef-kube-api-access-tvfng\" (UniqueName: \"kubernetes.io/projected/cf87280a-7926-4446-85f5-294bf2a519ef-kube-api-access-tvfng\") pod \"cf87280a-7926-4446-85f5-294bf2a519ef\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") "
Oct 29 23:29:42.500276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.500243    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-xks-ca\") pod \"cf87280a-7926-4446-85f5-294bf2a519ef\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") "
Oct 29 23:29:42.500349 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.500291    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-openshift-ca\") pod \"cf87280a-7926-4446-85f5-294bf2a519ef\" (UID: \"cf87280a-7926-4446-85f5-294bf2a519ef\") "
Oct 29 23:29:42.500665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.500637    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-openshift-ca" pod "cf87280a-7926-4446-85f5-294bf2a519ef" (UID: "cf87280a-7926-4446-85f5-294bf2a519ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:42.502213 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.502184    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf87280a-7926-4446-85f5-294bf2a519ef-kube-api-access-tvfng" pod "cf87280a-7926-4446-85f5-294bf2a519ef" (UID: "cf87280a-7926-4446-85f5-294bf2a519ef"). InnerVolumeSpecName "kube-api-access-tvfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:42.502314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.502262    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-xks-ca" pod "cf87280a-7926-4446-85f5-294bf2a519ef" (UID: "cf87280a-7926-4446-85f5-294bf2a519ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:42.556001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.546444    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x" podStartSLOduration=1.546421757 podStartE2EDuration="1.546421757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:42.348220886 +0000 UTC m=+960.452481273" watchObservedRunningTime="2026-10-29 23:29:42.546421757 +0000 UTC m=+960.650682144"
Oct 29 23:29:42.602124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.602089    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvfng\" (UniqueName: \"kubernetes.io/projected/cf87280a-7926-4446-85f5-294bf2a519ef-kube-api-access-tvfng\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:42.602124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.602118    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:42.602124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:42.602128    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf87280a-7926-4446-85f5-294bf2a519ef-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:43.157379 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:43.157296    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 23:29:43.157379 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:43.157353    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 23:29:43.296002 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:43.295971    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"
Oct 29 23:29:43.296002 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:43.295984    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh" event={"ID":"cf87280a-7926-4446-85f5-294bf2a519ef","Type":"ContainerDied","Data":"325c5713497107d1cb28fd4ab83046025e32810d9efa54227649dec3c417ad47"}
Oct 29 23:29:43.296529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:43.296052    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="325c5713497107d1cb28fd4ab83046025e32810d9efa54227649dec3c417ad47"
Oct 29 23:29:43.298053 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:43.298023    6203 generic.go:356] "Generic (PLEG): container finished" podID="3f9c42cf-9e27-4021-8682-3ca0e5d916f4" containerID="084e0be52fe399878629458d4fd18366e94a3f8ca7c1ca2cf2eceaa67ae7d19a" exitCode=0
Oct 29 23:29:43.298183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:43.298084    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x" event={"ID":"3f9c42cf-9e27-4021-8682-3ca0e5d916f4","Type":"ContainerDied","Data":"084e0be52fe399878629458d4fd18366e94a3f8ca7c1ca2cf2eceaa67ae7d19a"}
Oct 29 23:29:44.160348 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.160307    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 23:29:44.160624 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.160593    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 23:29:44.610738 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.610709    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:44.642904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.628125    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-openshift-ca\") pod \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") "
Oct 29 23:29:44.642904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.628255    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-kube-api-access-sjvn2\" (UniqueName: \"kubernetes.io/projected/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-kube-api-access-sjvn2\") pod \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") "
Oct 29 23:29:44.642904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.628418    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-xks-ca\") pod \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\" (UID: \"3f9c42cf-9e27-4021-8682-3ca0e5d916f4\") "
Oct 29 23:29:44.651467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.651097    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-openshift-ca" pod "3f9c42cf-9e27-4021-8682-3ca0e5d916f4" (UID: "3f9c42cf-9e27-4021-8682-3ca0e5d916f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:44.674899 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.672166    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-kube-api-access-sjvn2" pod "3f9c42cf-9e27-4021-8682-3ca0e5d916f4" (UID: "3f9c42cf-9e27-4021-8682-3ca0e5d916f4"). InnerVolumeSpecName "kube-api-access-sjvn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:44.675757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.675485    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-xks-ca" pod "3f9c42cf-9e27-4021-8682-3ca0e5d916f4" (UID: "3f9c42cf-9e27-4021-8682-3ca0e5d916f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:44.738107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.732737    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:44.738107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.732777    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:44.738107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:44.732794    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvn2\" (UniqueName: \"kubernetes.io/projected/3f9c42cf-9e27-4021-8682-3ca0e5d916f4-kube-api-access-sjvn2\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:45.157658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:45.157558    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 23:29:45.157658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:45.157630    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 23:29:45.318334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:45.318297    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"
Oct 29 23:29:45.318334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:45.318320    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x" event={"ID":"3f9c42cf-9e27-4021-8682-3ca0e5d916f4","Type":"ContainerDied","Data":"4fe3ed72bbee0d84a35585b0fab7e481c8978dd7a14f2ebd4eb19608ad24f95a"}
Oct 29 23:29:45.318543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:45.318375    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe3ed72bbee0d84a35585b0fab7e481c8978dd7a14f2ebd4eb19608ad24f95a"
Oct 29 23:29:46.157500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.157457    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 29 23:29:46.157931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.157535    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 29 23:29:46.596578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.596536    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"]
Oct 29 23:29:46.597155 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.597135    6203 cpu_manager.go:414] "containerMap: removing container" podUID="3f9c42cf-9e27-4021-8682-3ca0e5d916f4" containerName="revoke-keys"
Oct 29 23:29:46.597245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.597155    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9c42cf-9e27-4021-8682-3ca0e5d916f4" containerName="revoke-keys"
Oct 29 23:29:46.597245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.597184    6203 cpu_manager.go:414] "containerMap: removing container" podUID="cf87280a-7926-4446-85f5-294bf2a519ef" containerName="revoke-keys"
Oct 29 23:29:46.597245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.597192    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf87280a-7926-4446-85f5-294bf2a519ef" containerName="revoke-keys"
Oct 29 23:29:46.597392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.597337    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9c42cf-9e27-4021-8682-3ca0e5d916f4" containerName="revoke-keys"
Oct 29 23:29:46.597392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.597354    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf87280a-7926-4446-85f5-294bf2a519ef" containerName="revoke-keys"
Oct 29 23:29:46.601142 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.601120    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.621656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.621598    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"]
Oct 29 23:29:46.633794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.633764    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:29:46.633957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.633918    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:29:46.638051 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.638034    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:29:46.638122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.638033    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:29:46.669364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.669334    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.669535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.669492    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.669593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.669551    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ft9\" (UniqueName: \"kubernetes.io/projected/f7029084-02e0-48d9-a3bc-4f67d12c4297-kube-api-access-x7ft9\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.771235 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.771196    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.771381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.771259    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ft9\" (UniqueName: \"kubernetes.io/projected/f7029084-02e0-48d9-a3bc-4f67d12c4297-kube-api-access-x7ft9\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.771381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.771346    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.772025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.771998    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.773774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.773740    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.800459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.800428    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ft9\" (UniqueName: \"kubernetes.io/projected/f7029084-02e0-48d9-a3bc-4f67d12c4297-kube-api-access-x7ft9\") pod \"maas-api-revoke-sub-64b99370-bdmj5\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:46.915702 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:46.915623    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:47.236582 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.236548    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"]
Oct 29 23:29:47.238180 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:47.238146    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7029084_02e0_48d9_a3bc_4f67d12c4297.slice/crio-6df006471ef5ffdf3c863351251ce3c8637c1f7cf6520f689ec54cdc81eaa8f4 WatchSource:0}: Error finding container 6df006471ef5ffdf3c863351251ce3c8637c1f7cf6520f689ec54cdc81eaa8f4: Status 404 returned error can't find the container with id 6df006471ef5ffdf3c863351251ce3c8637c1f7cf6520f689ec54cdc81eaa8f4
Oct 29 23:29:47.339915 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.337489    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5" event={"ID":"f7029084-02e0-48d9-a3bc-4f67d12c4297","Type":"ContainerStarted","Data":"6df006471ef5ffdf3c863351251ce3c8637c1f7cf6520f689ec54cdc81eaa8f4"}
Oct 29 23:29:47.420497 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.420465    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:47.420941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.420912    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:47.421704 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.421682    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:29:47.569602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.569515    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"]
Oct 29 23:29:47.575230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.575204    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.600645 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.600616    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"]
Oct 29 23:29:47.682424 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.682378    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.682591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.682434    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.682591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.682520    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2kks\" (UniqueName: \"kubernetes.io/projected/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-kube-api-access-m2kks\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.783516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.783475    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.783685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.783530    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.783685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.783553    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2kks\" (UniqueName: \"kubernetes.io/projected/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-kube-api-access-m2kks\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.784322 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.784289    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.786126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.786103    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.809493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.809468    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2kks\" (UniqueName: \"kubernetes.io/projected/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-kube-api-access-m2kks\") pod \"maas-api-revoke-sub-b9dc6892-59mzm\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:47.891639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:47.891551    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:48.211190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:48.211147    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"]
Oct 29 23:29:48.349253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:48.349217    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm" event={"ID":"8c1c969c-0ca7-4083-b33d-fd81bdc20aad","Type":"ContainerStarted","Data":"f1b2b29cda4f3dd5866ed562e1441b2d8cbe47dec442bfb9c5436212d4883610"}
Oct 29 23:29:48.351027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:48.350992    6203 generic.go:356] "Generic (PLEG): container finished" podID="f7029084-02e0-48d9-a3bc-4f67d12c4297" containerID="4344196f3455dbe86f33e449495a004a7eec5c1e134b1a95a6804ecf8576afde" exitCode=0
Oct 29 23:29:48.351657 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:48.351593    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5" event={"ID":"f7029084-02e0-48d9-a3bc-4f67d12c4297","Type":"ContainerDied","Data":"4344196f3455dbe86f33e449495a004a7eec5c1e134b1a95a6804ecf8576afde"}
Oct 29 23:29:49.358622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.358533    6203 generic.go:356] "Generic (PLEG): container finished" podID="8c1c969c-0ca7-4083-b33d-fd81bdc20aad" containerID="6c81a697b56b31c8c1df032108147f7330349db481c40a2c17392cd6c53d78c8" exitCode=0
Oct 29 23:29:49.358622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.358601    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm" event={"ID":"8c1c969c-0ca7-4083-b33d-fd81bdc20aad","Type":"ContainerDied","Data":"6c81a697b56b31c8c1df032108147f7330349db481c40a2c17392cd6c53d78c8"}
Oct 29 23:29:49.497048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.497024    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:49.604953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.604921    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-openshift-ca\") pod \"f7029084-02e0-48d9-a3bc-4f67d12c4297\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") "
Oct 29 23:29:49.605117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.605045    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-xks-ca\") pod \"f7029084-02e0-48d9-a3bc-4f67d12c4297\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") "
Oct 29 23:29:49.605177 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.605160    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7029084-02e0-48d9-a3bc-4f67d12c4297-kube-api-access-x7ft9\" (UniqueName: \"kubernetes.io/projected/f7029084-02e0-48d9-a3bc-4f67d12c4297-kube-api-access-x7ft9\") pod \"f7029084-02e0-48d9-a3bc-4f67d12c4297\" (UID: \"f7029084-02e0-48d9-a3bc-4f67d12c4297\") "
Oct 29 23:29:49.605328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.605297    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-openshift-ca" pod "f7029084-02e0-48d9-a3bc-4f67d12c4297" (UID: "f7029084-02e0-48d9-a3bc-4f67d12c4297"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:49.605650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.605624    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:49.607592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.607550    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-xks-ca" pod "f7029084-02e0-48d9-a3bc-4f67d12c4297" (UID: "f7029084-02e0-48d9-a3bc-4f67d12c4297"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:49.607716 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.607683    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7029084-02e0-48d9-a3bc-4f67d12c4297-kube-api-access-x7ft9" pod "f7029084-02e0-48d9-a3bc-4f67d12c4297" (UID: "f7029084-02e0-48d9-a3bc-4f67d12c4297"). InnerVolumeSpecName "kube-api-access-x7ft9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:49.706461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.706428    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ft9\" (UniqueName: \"kubernetes.io/projected/f7029084-02e0-48d9-a3bc-4f67d12c4297-kube-api-access-x7ft9\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:49.706461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:49.706455    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7029084-02e0-48d9-a3bc-4f67d12c4297-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:50.189132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.189100    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"]
Oct 29 23:29:50.189650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.189636    6203 cpu_manager.go:414] "containerMap: removing container" podUID="f7029084-02e0-48d9-a3bc-4f67d12c4297" containerName="revoke-keys"
Oct 29 23:29:50.189690 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.189651    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7029084-02e0-48d9-a3bc-4f67d12c4297" containerName="revoke-keys"
Oct 29 23:29:50.189736 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.189726    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7029084-02e0-48d9-a3bc-4f67d12c4297" containerName="revoke-keys"
Oct 29 23:29:50.193125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.193103    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.221157 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.221129    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"]
Oct 29 23:29:50.313755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.313659    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.313931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.313855    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwp7\" (UniqueName: \"kubernetes.io/projected/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-kube-api-access-5xwp7\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.313931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.313908    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.365316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.365282    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5" event={"ID":"f7029084-02e0-48d9-a3bc-4f67d12c4297","Type":"ContainerDied","Data":"6df006471ef5ffdf3c863351251ce3c8637c1f7cf6520f689ec54cdc81eaa8f4"}
Oct 29 23:29:50.365698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.365319    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"
Oct 29 23:29:50.365698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.365338    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df006471ef5ffdf3c863351251ce3c8637c1f7cf6520f689ec54cdc81eaa8f4"
Oct 29 23:29:50.415121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.415080    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwp7\" (UniqueName: \"kubernetes.io/projected/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-kube-api-access-5xwp7\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.415121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.415125    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.415320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.415177    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.415842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.415821    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.417775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.417755    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.441174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.441149    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwp7\" (UniqueName: \"kubernetes.io/projected/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-kube-api-access-5xwp7\") pod \"maas-api-revoke-sub-bce274a6-rt7f9\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.505544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.505518    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:50.510985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.510964    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:50.620311 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.620282    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-openshift-ca\") pod \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") "
Oct 29 23:29:50.620476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.620453    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-kube-api-access-m2kks\" (UniqueName: \"kubernetes.io/projected/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-kube-api-access-m2kks\") pod \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") "
Oct 29 23:29:50.620523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.620504    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-xks-ca\") pod \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\" (UID: \"8c1c969c-0ca7-4083-b33d-fd81bdc20aad\") "
Oct 29 23:29:50.620671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.620652    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-openshift-ca" pod "8c1c969c-0ca7-4083-b33d-fd81bdc20aad" (UID: "8c1c969c-0ca7-4083-b33d-fd81bdc20aad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:50.620923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.620900    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:50.622596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.622562    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-xks-ca" pod "8c1c969c-0ca7-4083-b33d-fd81bdc20aad" (UID: "8c1c969c-0ca7-4083-b33d-fd81bdc20aad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:50.622705 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.622578    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-kube-api-access-m2kks" pod "8c1c969c-0ca7-4083-b33d-fd81bdc20aad" (UID: "8c1c969c-0ca7-4083-b33d-fd81bdc20aad"). InnerVolumeSpecName "kube-api-access-m2kks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:50.721860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.721823    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2kks\" (UniqueName: \"kubernetes.io/projected/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-kube-api-access-m2kks\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:50.721860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.721856    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1c969c-0ca7-4083-b33d-fd81bdc20aad-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:50.811745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:50.811714    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"]
Oct 29 23:29:50.812004 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:50.811982    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6e1fab_faa1_44d6_bfaa_2e0ac80317bd.slice/crio-351640aff2a9cfa192b3400fb0dd5c52576891d5666f69478b53c1115b0f4eb9 WatchSource:0}: Error finding container 351640aff2a9cfa192b3400fb0dd5c52576891d5666f69478b53c1115b0f4eb9: Status 404 returned error can't find the container with id 351640aff2a9cfa192b3400fb0dd5c52576891d5666f69478b53c1115b0f4eb9
Oct 29 23:29:51.372675 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:51.372645    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"
Oct 29 23:29:51.373107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:51.372684    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm" event={"ID":"8c1c969c-0ca7-4083-b33d-fd81bdc20aad","Type":"ContainerDied","Data":"f1b2b29cda4f3dd5866ed562e1441b2d8cbe47dec442bfb9c5436212d4883610"}
Oct 29 23:29:51.373107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:51.372738    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b2b29cda4f3dd5866ed562e1441b2d8cbe47dec442bfb9c5436212d4883610"
Oct 29 23:29:51.374407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:51.374381    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9" event={"ID":"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd","Type":"ContainerStarted","Data":"ad72fa685023ca43caa54b35b5dbc3a36c09921733e62721e8f4c58ec3606446"}
Oct 29 23:29:51.374524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:51.374425    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9" event={"ID":"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd","Type":"ContainerStarted","Data":"351640aff2a9cfa192b3400fb0dd5c52576891d5666f69478b53c1115b0f4eb9"}
Oct 29 23:29:51.422960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:51.422909    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9" podStartSLOduration=1.422895735 podStartE2EDuration="1.422895735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:51.422318687 +0000 UTC m=+969.526579073" watchObservedRunningTime="2026-10-29 23:29:51.422895735 +0000 UTC m=+969.527156114"
Oct 29 23:29:52.382940 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.382898    6203 generic.go:356] "Generic (PLEG): container finished" podID="5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" containerID="ad72fa685023ca43caa54b35b5dbc3a36c09921733e62721e8f4c58ec3606446" exitCode=0
Oct 29 23:29:52.383405 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.382977    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9" event={"ID":"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd","Type":"ContainerDied","Data":"ad72fa685023ca43caa54b35b5dbc3a36c09921733e62721e8f4c58ec3606446"}
Oct 29 23:29:52.802531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.802491    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"]
Oct 29 23:29:52.803185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.803163    6203 cpu_manager.go:414] "containerMap: removing container" podUID="8c1c969c-0ca7-4083-b33d-fd81bdc20aad" containerName="revoke-keys"
Oct 29 23:29:52.803185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.803184    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1c969c-0ca7-4083-b33d-fd81bdc20aad" containerName="revoke-keys"
Oct 29 23:29:52.803370 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.803357    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1c969c-0ca7-4083-b33d-fd81bdc20aad" containerName="revoke-keys"
Oct 29 23:29:52.806974 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.806952    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:52.831252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.831224    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"]
Oct 29 23:29:52.956781 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.956751    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlrt\" (UniqueName: \"kubernetes.io/projected/e0e27aa1-da0e-412d-9f54-c0b3289fc469-kube-api-access-zjlrt\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:52.956999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.956797    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:52.957048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:52.957018    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.058364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.058275    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlrt\" (UniqueName: \"kubernetes.io/projected/e0e27aa1-da0e-412d-9f54-c0b3289fc469-kube-api-access-zjlrt\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.058364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.058321    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.058604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.058413    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.059061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.059034    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.060937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.060912    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.087021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.086993    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlrt\" (UniqueName: \"kubernetes.io/projected/e0e27aa1-da0e-412d-9f54-c0b3289fc469-kube-api-access-zjlrt\") pod \"maas-api-revoke-sub-c2fb3083-fmktt\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.119620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.119594    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:53.437216 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:53.437186    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e27aa1_da0e_412d_9f54_c0b3289fc469.slice/crio-20d7e4859b5003608ee851672b3d208139992172a23b1f49cc3df575f49b1fb3 WatchSource:0}: Error finding container 20d7e4859b5003608ee851672b3d208139992172a23b1f49cc3df575f49b1fb3: Status 404 returned error can't find the container with id 20d7e4859b5003608ee851672b3d208139992172a23b1f49cc3df575f49b1fb3
Oct 29 23:29:53.437668 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.437642    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"]
Oct 29 23:29:53.542406 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.542383    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:53.664833 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.664799    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"]
Oct 29 23:29:53.665122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665101    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-openshift-ca\") pod \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") "
Oct 29 23:29:53.665262 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665247    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-kube-api-access-5xwp7\" (UniqueName: \"kubernetes.io/projected/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-kube-api-access-5xwp7\") pod \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") "
Oct 29 23:29:53.665359 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665346    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-xks-ca\") pod \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\" (UID: \"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd\") "
Oct 29 23:29:53.665500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665480    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" containerName="revoke-keys"
Oct 29 23:29:53.665500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665488    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-openshift-ca" pod "5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" (UID: "5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:53.665573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665506    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" containerName="revoke-keys"
Oct 29 23:29:53.665700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665674    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" containerName="revoke-keys"
Oct 29 23:29:53.665825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.665809    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.667471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.667442    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-xks-ca" pod "5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" (UID: "5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:53.667545 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.667484    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-kube-api-access-5xwp7" pod "5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" (UID: "5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd"). InnerVolumeSpecName "kube-api-access-5xwp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:53.669138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.669117    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.688381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.688361    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2a53c9-openshift-default-dockercfg-8qsfl\""
Oct 29 23:29:53.697429 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.697405    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"]
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767360    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767411    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767487    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767514    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767592    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767688    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767725    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767792    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrxg\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-kube-api-access-fvrxg\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767825    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767918    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.769698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.767936    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwp7\" (UniqueName: \"kubernetes.io/projected/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd-kube-api-access-5xwp7\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:53.820027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.819992    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"]
Oct 29 23:29:53.823967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.823949    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.869393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.869355    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.869393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.869397    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.869617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.869459    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.869617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.869532    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.869617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.869569    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.869902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.869647    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrxg\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-kube-api-access-fvrxg\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870408 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.870384    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.870141    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.870349    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.869848    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.870383    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.870416    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.870508    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.870734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.870533    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.872849 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.872823    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.873170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.873151    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.892305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.889511    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"]
Oct 29 23:29:53.907084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.907005    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.907214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.907107    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrxg\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-kube-api-access-fvrxg\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:53.971677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.971641    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.971677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.971681    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.971937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.971701    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.971937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.971820    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.971937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.971871    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.972069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.971965    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpmt\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-kube-api-access-zmpmt\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.972069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.971995    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.972069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.972045    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.972182 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.972099    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:53.983147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:53.983118    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:54.060047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.060006    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"]
Oct 29 23:29:54.067073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.067040    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.077542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077508    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.077689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077567    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.077689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077596    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.077689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077639    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.077835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077696    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.077835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077753    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpmt\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-kube-api-access-zmpmt\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.077835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077801    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.078007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077861    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.078007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.077914    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.078099 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.078049    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.078492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.078463    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.078791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.078738    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.079455 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.079418    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.080253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.080181    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.082287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.082263    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.083642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.083597    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.093124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.093013    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"]
Oct 29 23:29:54.115611 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.115578    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.115808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.115784    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpmt\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-kube-api-access-zmpmt\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.138095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.138068    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:54.180212 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.179914    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.180212 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.180031    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gcv\" (UniqueName: \"kubernetes.io/projected/2df1bfda-c485-4595-a210-6a5e28a80e7f-kube-api-access-h5gcv\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.180212 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.180074    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.281442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.281402    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gcv\" (UniqueName: \"kubernetes.io/projected/2df1bfda-c485-4595-a210-6a5e28a80e7f-kube-api-access-h5gcv\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.281604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.281469    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.281690 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.281669    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.282301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.282280    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.283852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.283831    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.292437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.292415    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"]
Oct 29 23:29:54.292522 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:54.292433    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc61897c_8bd9_48f7_97eb_2a4e277c72f4.slice/crio-73979831a9fc748226275b7cf0ec162a3e0565bcff3d6dc435a0dd6fec6844a3 WatchSource:0}: Error finding container 73979831a9fc748226275b7cf0ec162a3e0565bcff3d6dc435a0dd6fec6844a3: Status 404 returned error can't find the container with id 73979831a9fc748226275b7cf0ec162a3e0565bcff3d6dc435a0dd6fec6844a3
Oct 29 23:29:54.295029 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.294995    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:54.295113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.295059    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:54.295113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.295086    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:54.308670 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.308627    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gcv\" (UniqueName: \"kubernetes.io/projected/2df1bfda-c485-4595-a210-6a5e28a80e7f-kube-api-access-h5gcv\") pod \"maas-api-revoke-sub-d891a57f-mwn8g\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.401897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.400856    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" event={"ID":"bc61897c-8bd9-48f7-97eb-2a4e277c72f4","Type":"ContainerStarted","Data":"6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135"}
Oct 29 23:29:54.401897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.400954    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" event={"ID":"bc61897c-8bd9-48f7-97eb-2a4e277c72f4","Type":"ContainerStarted","Data":"73979831a9fc748226275b7cf0ec162a3e0565bcff3d6dc435a0dd6fec6844a3"}
Oct 29 23:29:54.403012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.402981    6203 generic.go:356] "Generic (PLEG): container finished" podID="e0e27aa1-da0e-412d-9f54-c0b3289fc469" containerID="a687a9996c1376b3a03e3b7af5de973615c49f37c9dc496e59d7df17d48554e3" exitCode=0
Oct 29 23:29:54.403134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.403048    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt" event={"ID":"e0e27aa1-da0e-412d-9f54-c0b3289fc469","Type":"ContainerDied","Data":"a687a9996c1376b3a03e3b7af5de973615c49f37c9dc496e59d7df17d48554e3"}
Oct 29 23:29:54.403134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.403127    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt" event={"ID":"e0e27aa1-da0e-412d-9f54-c0b3289fc469","Type":"ContainerStarted","Data":"20d7e4859b5003608ee851672b3d208139992172a23b1f49cc3df575f49b1fb3"}
Oct 29 23:29:54.409304 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.409265    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9" event={"ID":"5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd","Type":"ContainerDied","Data":"351640aff2a9cfa192b3400fb0dd5c52576891d5666f69478b53c1115b0f4eb9"}
Oct 29 23:29:54.409304 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.409287    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"
Oct 29 23:29:54.409490 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.409313    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351640aff2a9cfa192b3400fb0dd5c52576891d5666f69478b53c1115b0f4eb9"
Oct 29 23:29:54.413970 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.413947    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:54.463420 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.463120    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"]
Oct 29 23:29:54.466273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.466190    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podStartSLOduration=1.466164802 podStartE2EDuration="1.466164802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:54.461322416 +0000 UTC m=+972.565582803" watchObservedRunningTime="2026-10-29 23:29:54.466164802 +0000 UTC m=+972.570425187"
Oct 29 23:29:54.475272 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:54.475220    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7aa6a8_95b4_4b68_b1a2_4a91b285a280.slice/crio-613126627ff12e7796ced059c74a42f5891d3947f4adcb0eb99fdb0c1143b31f WatchSource:0}: Error finding container 613126627ff12e7796ced059c74a42f5891d3947f4adcb0eb99fdb0c1143b31f: Status 404 returned error can't find the container with id 613126627ff12e7796ced059c74a42f5891d3947f4adcb0eb99fdb0c1143b31f
Oct 29 23:29:54.479078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.479042    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:54.479147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.479127    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:54.479185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.479169    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:29:54.726567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.726537    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"]
Oct 29 23:29:54.727273 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:29:54.727248    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df1bfda_c485_4595_a210_6a5e28a80e7f.slice/crio-edeed2b85927c5cbf3b308bbb4de4cb3a6b7d6f2aab0b375d9c56986c1229208 WatchSource:0}: Error finding container edeed2b85927c5cbf3b308bbb4de4cb3a6b7d6f2aab0b375d9c56986c1229208: Status 404 returned error can't find the container with id edeed2b85927c5cbf3b308bbb4de4cb3a6b7d6f2aab0b375d9c56986c1229208
Oct 29 23:29:54.986015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:54.984001    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:29:55.416993 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.416966    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" event={"ID":"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280","Type":"ContainerStarted","Data":"63f9acdeebc29ec5539fbb6c1926360856b782ac720a83f6539b812f846e9091"}
Oct 29 23:29:55.417149 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.417016    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" event={"ID":"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280","Type":"ContainerStarted","Data":"613126627ff12e7796ced059c74a42f5891d3947f4adcb0eb99fdb0c1143b31f"}
Oct 29 23:29:55.418317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.418288    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g" event={"ID":"2df1bfda-c485-4595-a210-6a5e28a80e7f","Type":"ContainerStarted","Data":"a7786a390e37b385949ef1cc5e1cefedf902e25db33173a10be958d2b9b67d7b"}
Oct 29 23:29:55.418440 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.418330    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g" event={"ID":"2df1bfda-c485-4595-a210-6a5e28a80e7f","Type":"ContainerStarted","Data":"edeed2b85927c5cbf3b308bbb4de4cb3a6b7d6f2aab0b375d9c56986c1229208"}
Oct 29 23:29:55.487451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.487396    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podStartSLOduration=2.487382338 podStartE2EDuration="2.487382338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:55.486202212 +0000 UTC m=+973.590462598" watchObservedRunningTime="2026-10-29 23:29:55.487382338 +0000 UTC m=+973.591642792"
Oct 29 23:29:55.587945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.587033    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g" podStartSLOduration=1.5870060289999999 podStartE2EDuration="1.587006029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:29:55.572655123 +0000 UTC m=+973.676915509" watchObservedRunningTime="2026-10-29 23:29:55.587006029 +0000 UTC m=+973.691266416"
Oct 29 23:29:55.706715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.706679    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:55.804452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.804407    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-xks-ca\") pod \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") "
Oct 29 23:29:55.804616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.804523    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-openshift-ca\") pod \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") "
Oct 29 23:29:55.804616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.804580    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e27aa1-da0e-412d-9f54-c0b3289fc469-kube-api-access-zjlrt\" (UniqueName: \"kubernetes.io/projected/e0e27aa1-da0e-412d-9f54-c0b3289fc469-kube-api-access-zjlrt\") pod \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\" (UID: \"e0e27aa1-da0e-412d-9f54-c0b3289fc469\") "
Oct 29 23:29:55.806247 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.806216    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-openshift-ca" pod "e0e27aa1-da0e-412d-9f54-c0b3289fc469" (UID: "e0e27aa1-da0e-412d-9f54-c0b3289fc469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:55.812470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.812437    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-xks-ca" pod "e0e27aa1-da0e-412d-9f54-c0b3289fc469" (UID: "e0e27aa1-da0e-412d-9f54-c0b3289fc469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:55.812770 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.812750    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e27aa1-da0e-412d-9f54-c0b3289fc469-kube-api-access-zjlrt" pod "e0e27aa1-da0e-412d-9f54-c0b3289fc469" (UID: "e0e27aa1-da0e-412d-9f54-c0b3289fc469"). InnerVolumeSpecName "kube-api-access-zjlrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:55.908539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.907522    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjlrt\" (UniqueName: \"kubernetes.io/projected/e0e27aa1-da0e-412d-9f54-c0b3289fc469-kube-api-access-zjlrt\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:55.908539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.907554    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:55.908539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.907568    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e27aa1-da0e-412d-9f54-c0b3289fc469-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:55.985798 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.985692    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 23:29:55.985798 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:55.985759    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 23:29:56.146901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.139174    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:29:56.146901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.142970    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 23:29:56.146901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.143023    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 23:29:56.445451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.445407    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt" event={"ID":"e0e27aa1-da0e-412d-9f54-c0b3289fc469","Type":"ContainerDied","Data":"20d7e4859b5003608ee851672b3d208139992172a23b1f49cc3df575f49b1fb3"}
Oct 29 23:29:56.467519 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.467484    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d7e4859b5003608ee851672b3d208139992172a23b1f49cc3df575f49b1fb3"
Oct 29 23:29:56.482903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.454069    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"
Oct 29 23:29:56.485803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.485768    6203 generic.go:356] "Generic (PLEG): container finished" podID="2df1bfda-c485-4595-a210-6a5e28a80e7f" containerID="a7786a390e37b385949ef1cc5e1cefedf902e25db33173a10be958d2b9b67d7b" exitCode=0
Oct 29 23:29:56.485996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.485969    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g" event={"ID":"2df1bfda-c485-4595-a210-6a5e28a80e7f","Type":"ContainerDied","Data":"a7786a390e37b385949ef1cc5e1cefedf902e25db33173a10be958d2b9b67d7b"}
Oct 29 23:29:56.984333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.984296    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 23:29:56.984748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:56.984355    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 23:29:57.144904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.142076    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 23:29:57.144904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.142140    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 23:29:57.806011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.805933    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:57.897939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.896605    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-xks-ca\") pod \"2df1bfda-c485-4595-a210-6a5e28a80e7f\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") "
Oct 29 23:29:57.897939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.896789    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df1bfda-c485-4595-a210-6a5e28a80e7f-kube-api-access-h5gcv\" (UniqueName: \"kubernetes.io/projected/2df1bfda-c485-4595-a210-6a5e28a80e7f-kube-api-access-h5gcv\") pod \"2df1bfda-c485-4595-a210-6a5e28a80e7f\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") "
Oct 29 23:29:57.897939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.896849    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-openshift-ca\") pod \"2df1bfda-c485-4595-a210-6a5e28a80e7f\" (UID: \"2df1bfda-c485-4595-a210-6a5e28a80e7f\") "
Oct 29 23:29:57.897939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.897591    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-openshift-ca" pod "2df1bfda-c485-4595-a210-6a5e28a80e7f" (UID: "2df1bfda-c485-4595-a210-6a5e28a80e7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:29:57.899343 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.899311    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-xks-ca" pod "2df1bfda-c485-4595-a210-6a5e28a80e7f" (UID: "2df1bfda-c485-4595-a210-6a5e28a80e7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:29:57.901578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.900429    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df1bfda-c485-4595-a210-6a5e28a80e7f-kube-api-access-h5gcv" pod "2df1bfda-c485-4595-a210-6a5e28a80e7f" (UID: "2df1bfda-c485-4595-a210-6a5e28a80e7f"). InnerVolumeSpecName "kube-api-access-h5gcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:29:57.984023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.983980    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 23:29:57.984213 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.984041    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 23:29:57.998169 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.998139    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5gcv\" (UniqueName: \"kubernetes.io/projected/2df1bfda-c485-4595-a210-6a5e28a80e7f-kube-api-access-h5gcv\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:57.998169 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.998166    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:57.998169 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:57.998177    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1bfda-c485-4595-a210-6a5e28a80e7f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:29:58.149962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:58.145002    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 23:29:58.149962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:58.145066    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 23:29:58.499537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:58.499507    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"
Oct 29 23:29:58.499813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:58.499786    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g" event={"ID":"2df1bfda-c485-4595-a210-6a5e28a80e7f","Type":"ContainerDied","Data":"edeed2b85927c5cbf3b308bbb4de4cb3a6b7d6f2aab0b375d9c56986c1229208"}
Oct 29 23:29:58.499864 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:58.499834    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edeed2b85927c5cbf3b308bbb4de4cb3a6b7d6f2aab0b375d9c56986c1229208"
Oct 29 23:29:58.984436 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:58.984392    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 23:29:58.984637 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:58.984448    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 23:29:59.142290 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:59.142250    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 23:29:59.142906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:59.142848    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 23:29:59.984211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:59.984172    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 23:29:59.984511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:29:59.984479    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 23:30:00.145906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.143373    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 23:30:00.145906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.143433    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 23:30:00.232765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.232727    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"]
Oct 29 23:30:00.233963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.233936    6203 cpu_manager.go:414] "containerMap: removing container" podUID="e0e27aa1-da0e-412d-9f54-c0b3289fc469" containerName="revoke-keys"
Oct 29 23:30:00.234110 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.234096    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e27aa1-da0e-412d-9f54-c0b3289fc469" containerName="revoke-keys"
Oct 29 23:30:00.234286 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.234240    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2df1bfda-c485-4595-a210-6a5e28a80e7f" containerName="revoke-keys"
Oct 29 23:30:00.234375 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.234362    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df1bfda-c485-4595-a210-6a5e28a80e7f" containerName="revoke-keys"
Oct 29 23:30:00.237899 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.234587    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e27aa1-da0e-412d-9f54-c0b3289fc469" containerName="revoke-keys"
Oct 29 23:30:00.238350 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.238332    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df1bfda-c485-4595-a210-6a5e28a80e7f" containerName="revoke-keys"
Oct 29 23:30:00.252000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.242692    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.254126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.253997    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"]
Oct 29 23:30:00.263204 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.263171    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:30:00.266558 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.264145    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:00.268956 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.268931    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"]
Oct 29 23:30:00.282863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.282117    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"]
Oct 29 23:30:00.282863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.282228    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.297385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.297357    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"]
Oct 29 23:30:00.303924 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.303866    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"]
Oct 29 23:30:00.334094 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.334021    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.334245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.334112    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qh8t\" (UniqueName: \"kubernetes.io/projected/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-kube-api-access-9qh8t\") pod \"maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.334245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.334155    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c1ce70-9018-4a6a-b823-3b497c0b6b47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:00.334376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.334248    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.334376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.334280    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5c6l\" (UniqueName: \"kubernetes.io/projected/18c1ce70-9018-4a6a-b823-3b497c0b6b47-kube-api-access-k5c6l\") pod \"maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:00.334477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.334384    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2dl\" (UniqueName: \"kubernetes.io/projected/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-kube-api-access-wj2dl\") pod \"maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.381249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.353273    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:30:00.417912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.396265    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:30:00.417912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.407123    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"]
Oct 29 23:30:00.473658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.435232    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5c6l\" (UniqueName: \"kubernetes.io/projected/18c1ce70-9018-4a6a-b823-3b497c0b6b47-kube-api-access-k5c6l\") pod \"maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:00.473658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.435335    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2dl\" (UniqueName: \"kubernetes.io/projected/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-kube-api-access-wj2dl\") pod \"maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.473658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.435404    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.473658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.435479    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qh8t\" (UniqueName: \"kubernetes.io/projected/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-kube-api-access-9qh8t\") pod \"maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.473658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.435514    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c1ce70-9018-4a6a-b823-3b497c0b6b47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:00.473658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.435598    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.473658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.436421    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.474274 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.474231    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c1ce70-9018-4a6a-b823-3b497c0b6b47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:00.475990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.475954    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.484720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.484625    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2dl\" (UniqueName: \"kubernetes.io/projected/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-kube-api-access-wj2dl\") pod \"maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.844362 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.828647    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qh8t\" (UniqueName: \"kubernetes.io/projected/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-kube-api-access-9qh8t\") pod \"maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.898911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.880190    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:00.984975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.889579    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5c6l\" (UniqueName: \"kubernetes.io/projected/18c1ce70-9018-4a6a-b823-3b497c0b6b47-kube-api-access-k5c6l\") pod \"maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:00.986132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.928170    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:00.992046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.985586    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 23:30:00.992272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:00.992246    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 23:30:01.039857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.039717    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:01.083903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.061609    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"]
Oct 29 23:30:01.135865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.135798    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.142834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.142712    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 23:30:01.142834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.142776    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 23:30:01.151331 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.150961    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:01.253168 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.253131    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.253436 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.253410    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lkc\" (UniqueName: \"kubernetes.io/projected/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-kube-api-access-t2lkc\") pod \"maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.264139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.264109    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"]
Oct 29 23:30:01.264372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.264354    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"]
Oct 29 23:30:01.264635 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.264607    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.387954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.371700    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vz2\" (UniqueName: \"kubernetes.io/projected/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-kube-api-access-h2vz2\") pod \"maas-api-key-cleanup-29888610-9qntv\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.387954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.380032    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.387954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.380126    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-9qntv\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.387954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.381115    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.387954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.381240    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lkc\" (UniqueName: \"kubernetes.io/projected/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-kube-api-access-t2lkc\") pod \"maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.446900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.442197    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lkc\" (UniqueName: \"kubernetes.io/projected/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-kube-api-access-t2lkc\") pod \"maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.512920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.488668    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vz2\" (UniqueName: \"kubernetes.io/projected/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-kube-api-access-h2vz2\") pod \"maas-api-key-cleanup-29888610-9qntv\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.512920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.488749    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-9qntv\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.512920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.489673    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888610-9qntv\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.550009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.549939    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vz2\" (UniqueName: \"kubernetes.io/projected/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-kube-api-access-h2vz2\") pod \"maas-api-key-cleanup-29888610-9qntv\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.584375 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.584082    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:01.677954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.676573    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:01.723926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.718871    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:01.938124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.914006    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:30:01.939585 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.938511    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:30:01.939585 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.938598    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:01.986919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.957711    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:01.986919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.958087    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:01.986919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.960530    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:01.995005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.993023    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 29 23:30:01.995005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:01.993103    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 29 23:30:02.188080 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:02.181238    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:02.196976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:02.193609    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 29 23:30:02.238995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:02.224284    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 23:30:02.238995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:02.224346    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 23:30:02.258019 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:02.250856    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:30:02.973865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:02.973801    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:03.004919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.003348    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 23:30:03.004919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.004706    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:30:03.004919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.004767    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:30:03.052570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.046061    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:03.055502 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.053055    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:30:03.201044 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.200996    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:03.223078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.223049    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:30:03.223334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.223314    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:30:03.568957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.561469    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"]
Oct 29 23:30:03.729056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.729009    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:03.752910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.737365    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" event={"ID":"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89","Type":"ContainerStarted","Data":"3eefdab6ccad8ba923fccbdca210ecb17c0bd31536473b773c45a82c26219e64"}
Oct 29 23:30:03.994685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:03.984434    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:04.053908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.043183    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 23:30:04.053908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.043120    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:04.068973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.054413    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:30:04.209906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.207967    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 29 23:30:04.312466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.232773    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:30:04.312466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.234831    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:04.312466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.258587    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:30:04.312466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.258661    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:04.312466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.266974    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:30:04.312466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.267039    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:04.604907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.590960    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 29 23:30:04.760917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:04.757238    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:05.023113 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:05.023082    6203 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.002s"
Oct 29 23:30:05.023366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.023353    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:30:05.023571 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.023559    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:30:05.026425 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.026395    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:30:05.026706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.026686    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:30:05.027344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.027318    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:30:05.027539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.027505    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:05.028904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.028513    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:05.056720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.054608    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:05.059256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.058595    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:05.067909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.065546    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:05.070707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.070670    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerStarted","Data":"24a162dffec81a646a53d4ec5a8ecdcf0202cf8758d4203dc6534d828bbcd6b7"}
Oct 29 23:30:05.109267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.109229    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:30:05.110527 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.110499    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:30:05.132114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.123326    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"]
Oct 29 23:30:05.146672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.144726    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"]
Oct 29 23:30:05.146672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.144800    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"]
Oct 29 23:30:05.146672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.144868    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:30:05.146672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.145846    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:30:05.151706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.149934    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"]
Oct 29 23:30:05.153823 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.152800    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" event={"ID":"18c1ce70-9018-4a6a-b823-3b497c0b6b47","Type":"ContainerStarted","Data":"5acd1d755c94f5d85e0532bcf6bc8f9b04ecebc7081cbcffb79910e7f0b5ab25"}
Oct 29 23:30:05.161781 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.161412    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" event={"ID":"7bc970d5-8060-4e34-bd76-ea0d0edf9b84","Type":"ContainerStarted","Data":"c2c055f969d80726c981db8a40eb6897dd12480d8b7ab025c67ef197a4f984d0"}
Oct 29 23:30:05.164549 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.164148    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" event={"ID":"a80b5e9e-d740-4a82-9c69-78ef95ba34c0","Type":"ContainerStarted","Data":"9c78ae7edd5494b89fac5464699f636d383ba26ecd3d3e1ad5319d56eb5aed49"}
Oct 29 23:30:05.173464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.173441    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:30:05.174499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.174476    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" event={"ID":"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89","Type":"ContainerStarted","Data":"0603c8d836ba4492348818eea8d0d6999fb8588eec7d88804f327ba4e3b4602b"}
Oct 29 23:30:05.209957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.209908    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:05.241843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.240217    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:30:05.246392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.246366    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:30:05.260921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.260895    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:30:05.643679 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:05.643634    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" podStartSLOduration=5.643615576 podStartE2EDuration="5.643615576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:05.641827834 +0000 UTC m=+983.746088221" watchObservedRunningTime="2026-10-29 23:30:05.643615576 +0000 UTC m=+983.747875963"
Oct 29 23:30:06.182301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.182267    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" event={"ID":"7bc970d5-8060-4e34-bd76-ea0d0edf9b84","Type":"ContainerStarted","Data":"cc6d2f9336b69f4c7895223f6a3f7df9a9425564eedcd5220e4a03dee5ceef33"}
Oct 29 23:30:06.183857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.183823    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" event={"ID":"a80b5e9e-d740-4a82-9c69-78ef95ba34c0","Type":"ContainerStarted","Data":"91c646f95e5cb1e229eec5f02ee7f91ed3211ceed2e96633f69f8f515561818f"}
Oct 29 23:30:06.185378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.185354    6203 generic.go:356] "Generic (PLEG): container finished" podID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerID="0603c8d836ba4492348818eea8d0d6999fb8588eec7d88804f327ba4e3b4602b" exitCode=60
Oct 29 23:30:06.185481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.185434    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" event={"ID":"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89","Type":"ContainerDied","Data":"0603c8d836ba4492348818eea8d0d6999fb8588eec7d88804f327ba4e3b4602b"}
Oct 29 23:30:06.185659 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.185643    6203 scope.go:122] "RemoveContainer" containerID="0603c8d836ba4492348818eea8d0d6999fb8588eec7d88804f327ba4e3b4602b"
Oct 29 23:30:06.187084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.187058    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerStarted","Data":"51bf8e440f67b4019fc5af739bd6104167e9577cdc485f5cf00e9db47b15175a"}
Oct 29 23:30:06.188503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.188480    6203 generic.go:356] "Generic (PLEG): container finished" podID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerID="ca084bb7072e8a8d666d1e5790922d031e34b3e4fc296fc781cd079173ac8d58" exitCode=60
Oct 29 23:30:06.188574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.188558    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" event={"ID":"18c1ce70-9018-4a6a-b823-3b497c0b6b47","Type":"ContainerDied","Data":"ca084bb7072e8a8d666d1e5790922d031e34b3e4fc296fc781cd079173ac8d58"}
Oct 29 23:30:06.188740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.188724    6203 scope.go:122] "RemoveContainer" containerID="ca084bb7072e8a8d666d1e5790922d031e34b3e4fc296fc781cd079173ac8d58"
Oct 29 23:30:06.281986 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.281931    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" podStartSLOduration=6.281915242 podStartE2EDuration="6.281915242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:06.231157497 +0000 UTC m=+984.335417886" watchObservedRunningTime="2026-10-29 23:30:06.281915242 +0000 UTC m=+984.386175631"
Oct 29 23:30:06.331769 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.331659    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" podStartSLOduration=6.331645072 podStartE2EDuration="6.331645072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:06.28418333 +0000 UTC m=+984.388443743" watchObservedRunningTime="2026-10-29 23:30:06.331645072 +0000 UTC m=+984.435905521"
Oct 29 23:30:06.379624 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:06.379563    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" podStartSLOduration=6.379544517 podStartE2EDuration="6.379544517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:06.376081237 +0000 UTC m=+984.480341621" watchObservedRunningTime="2026-10-29 23:30:06.379544517 +0000 UTC m=+984.483804906"
Oct 29 23:30:07.197563 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.197530    6203 generic.go:356] "Generic (PLEG): container finished" podID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerID="51bf8e440f67b4019fc5af739bd6104167e9577cdc485f5cf00e9db47b15175a" exitCode=60
Oct 29 23:30:07.197728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.197623    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerDied","Data":"51bf8e440f67b4019fc5af739bd6104167e9577cdc485f5cf00e9db47b15175a"}
Oct 29 23:30:07.197980 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.197962    6203 scope.go:122] "RemoveContainer" containerID="51bf8e440f67b4019fc5af739bd6104167e9577cdc485f5cf00e9db47b15175a"
Oct 29 23:30:07.199710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.199684    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" event={"ID":"18c1ce70-9018-4a6a-b823-3b497c0b6b47","Type":"ContainerStarted","Data":"bd7d73b7dc6400abb154a56298b3fb491d41cd548302c3195d7801dc17a461c3"}
Oct 29 23:30:07.201422 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.201400    6203 generic.go:356] "Generic (PLEG): container finished" podID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerID="cc6d2f9336b69f4c7895223f6a3f7df9a9425564eedcd5220e4a03dee5ceef33" exitCode=60
Oct 29 23:30:07.201504 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.201460    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" event={"ID":"7bc970d5-8060-4e34-bd76-ea0d0edf9b84","Type":"ContainerDied","Data":"cc6d2f9336b69f4c7895223f6a3f7df9a9425564eedcd5220e4a03dee5ceef33"}
Oct 29 23:30:07.201674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.201661    6203 scope.go:122] "RemoveContainer" containerID="cc6d2f9336b69f4c7895223f6a3f7df9a9425564eedcd5220e4a03dee5ceef33"
Oct 29 23:30:07.203093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.203074    6203 generic.go:356] "Generic (PLEG): container finished" podID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerID="91c646f95e5cb1e229eec5f02ee7f91ed3211ceed2e96633f69f8f515561818f" exitCode=60
Oct 29 23:30:07.203192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.203154    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" event={"ID":"a80b5e9e-d740-4a82-9c69-78ef95ba34c0","Type":"ContainerDied","Data":"91c646f95e5cb1e229eec5f02ee7f91ed3211ceed2e96633f69f8f515561818f"}
Oct 29 23:30:07.203359 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.203343    6203 scope.go:122] "RemoveContainer" containerID="91c646f95e5cb1e229eec5f02ee7f91ed3211ceed2e96633f69f8f515561818f"
Oct 29 23:30:07.205044 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.205024    6203 generic.go:356] "Generic (PLEG): container finished" podID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerID="b7afc30134ece2ed53a257a1e8428e8649703900206e2afc8e6172ff40e5f44d" exitCode=60
Oct 29 23:30:07.205131 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.205103    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" event={"ID":"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89","Type":"ContainerDied","Data":"b7afc30134ece2ed53a257a1e8428e8649703900206e2afc8e6172ff40e5f44d"}
Oct 29 23:30:07.205213 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.205151    6203 scope.go:122] "RemoveContainer" containerID="0603c8d836ba4492348818eea8d0d6999fb8588eec7d88804f327ba4e3b4602b"
Oct 29 23:30:07.205402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.205382    6203 scope.go:122] "RemoveContainer" containerID="b7afc30134ece2ed53a257a1e8428e8649703900206e2afc8e6172ff40e5f44d"
Oct 29 23:30:07.205640 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:07.205614    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb_odh-ai-gateway-infra(5a1f8c79-cf53-4cb9-8dad-c65fc4423c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89"
Oct 29 23:30:07.353258 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:07.353207    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" podStartSLOduration=7.353192033 podStartE2EDuration="7.353192033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:07.350980649 +0000 UTC m=+985.455241036" watchObservedRunningTime="2026-10-29 23:30:07.353192033 +0000 UTC m=+985.457452412"
Oct 29 23:30:08.215018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.214982    6203 generic.go:356] "Generic (PLEG): container finished" podID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerID="bd7d73b7dc6400abb154a56298b3fb491d41cd548302c3195d7801dc17a461c3" exitCode=60
Oct 29 23:30:08.215320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.215062    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" event={"ID":"18c1ce70-9018-4a6a-b823-3b497c0b6b47","Type":"ContainerDied","Data":"bd7d73b7dc6400abb154a56298b3fb491d41cd548302c3195d7801dc17a461c3"}
Oct 29 23:30:08.215320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.215143    6203 scope.go:122] "RemoveContainer" containerID="ca084bb7072e8a8d666d1e5790922d031e34b3e4fc296fc781cd079173ac8d58"
Oct 29 23:30:08.215510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.215490    6203 scope.go:122] "RemoveContainer" containerID="bd7d73b7dc6400abb154a56298b3fb491d41cd548302c3195d7801dc17a461c3"
Oct 29 23:30:08.215835 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:08.215808    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k_odh-ai-gateway-infra(18c1ce70-9018-4a6a-b823-3b497c0b6b47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47"
Oct 29 23:30:08.217664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.217637    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" event={"ID":"7bc970d5-8060-4e34-bd76-ea0d0edf9b84","Type":"ContainerStarted","Data":"c4704d052b931d21f8afee290c772d2ff6d1f93f3eb326ce30bda60db4bdc40c"}
Oct 29 23:30:08.220056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.220030    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" event={"ID":"a80b5e9e-d740-4a82-9c69-78ef95ba34c0","Type":"ContainerStarted","Data":"f3bee1555f3489ba3cdf74ae671e1841aa06672e05445c3c89a11ce54297b333"}
Oct 29 23:30:08.222664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.222621    6203 scope.go:122] "RemoveContainer" containerID="b7afc30134ece2ed53a257a1e8428e8649703900206e2afc8e6172ff40e5f44d"
Oct 29 23:30:08.223011 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:08.222982    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb_odh-ai-gateway-infra(5a1f8c79-cf53-4cb9-8dad-c65fc4423c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89"
Oct 29 23:30:08.224531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:08.224505    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerStarted","Data":"0d86cefa9029acf6e8e9bd6187503fae33d28e934bda39e45ef313089b1323aa"}
Oct 29 23:30:09.233480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.233450    6203 scope.go:122] "RemoveContainer" containerID="bd7d73b7dc6400abb154a56298b3fb491d41cd548302c3195d7801dc17a461c3"
Oct 29 23:30:09.233985 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:09.233764    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k_odh-ai-gateway-infra(18c1ce70-9018-4a6a-b823-3b497c0b6b47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47"
Oct 29 23:30:09.235509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.235479    6203 generic.go:356] "Generic (PLEG): container finished" podID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerID="c4704d052b931d21f8afee290c772d2ff6d1f93f3eb326ce30bda60db4bdc40c" exitCode=60
Oct 29 23:30:09.235656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.235568    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" event={"ID":"7bc970d5-8060-4e34-bd76-ea0d0edf9b84","Type":"ContainerDied","Data":"c4704d052b931d21f8afee290c772d2ff6d1f93f3eb326ce30bda60db4bdc40c"}
Oct 29 23:30:09.235656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.235628    6203 scope.go:122] "RemoveContainer" containerID="cc6d2f9336b69f4c7895223f6a3f7df9a9425564eedcd5220e4a03dee5ceef33"
Oct 29 23:30:09.235822 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.235807    6203 scope.go:122] "RemoveContainer" containerID="c4704d052b931d21f8afee290c772d2ff6d1f93f3eb326ce30bda60db4bdc40c"
Oct 29 23:30:09.236169 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:09.236137    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd_odh-ai-gateway-infra(7bc970d5-8060-4e34-bd76-ea0d0edf9b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84"
Oct 29 23:30:09.237694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.237673    6203 generic.go:356] "Generic (PLEG): container finished" podID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerID="f3bee1555f3489ba3cdf74ae671e1841aa06672e05445c3c89a11ce54297b333" exitCode=60
Oct 29 23:30:09.237793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.237737    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" event={"ID":"a80b5e9e-d740-4a82-9c69-78ef95ba34c0","Type":"ContainerDied","Data":"f3bee1555f3489ba3cdf74ae671e1841aa06672e05445c3c89a11ce54297b333"}
Oct 29 23:30:09.238038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.238022    6203 scope.go:122] "RemoveContainer" containerID="f3bee1555f3489ba3cdf74ae671e1841aa06672e05445c3c89a11ce54297b333"
Oct 29 23:30:09.238276 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:09.238251    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-9qntv_odh-ai-gateway-infra(a80b5e9e-d740-4a82-9c69-78ef95ba34c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0"
Oct 29 23:30:09.239679 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.239659    6203 generic.go:356] "Generic (PLEG): container finished" podID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerID="0d86cefa9029acf6e8e9bd6187503fae33d28e934bda39e45ef313089b1323aa" exitCode=60
Oct 29 23:30:09.239785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.239704    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerDied","Data":"0d86cefa9029acf6e8e9bd6187503fae33d28e934bda39e45ef313089b1323aa"}
Oct 29 23:30:09.240018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.240000    6203 scope.go:122] "RemoveContainer" containerID="0d86cefa9029acf6e8e9bd6187503fae33d28e934bda39e45ef313089b1323aa"
Oct 29 23:30:09.240283 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:09.240261    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl_odh-ai-gateway-infra(4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec"
Oct 29 23:30:09.248703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.248679    6203 scope.go:122] "RemoveContainer" containerID="91c646f95e5cb1e229eec5f02ee7f91ed3211ceed2e96633f69f8f515561818f"
Oct 29 23:30:09.261361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:09.261341    6203 scope.go:122] "RemoveContainer" containerID="51bf8e440f67b4019fc5af739bd6104167e9577cdc485f5cf00e9db47b15175a"
Oct 29 23:30:09.924798 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:09.924763    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:30:10.249110 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:10.249018    6203 scope.go:122] "RemoveContainer" containerID="c4704d052b931d21f8afee290c772d2ff6d1f93f3eb326ce30bda60db4bdc40c"
Oct 29 23:30:10.249546 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:10.249377    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd_odh-ai-gateway-infra(7bc970d5-8060-4e34-bd76-ea0d0edf9b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84"
Oct 29 23:30:10.250709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:10.250681    6203 scope.go:122] "RemoveContainer" containerID="f3bee1555f3489ba3cdf74ae671e1841aa06672e05445c3c89a11ce54297b333"
Oct 29 23:30:10.251023 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:10.251000    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-9qntv_odh-ai-gateway-infra(a80b5e9e-d740-4a82-9c69-78ef95ba34c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0"
Oct 29 23:30:10.252277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:10.252255    6203 scope.go:122] "RemoveContainer" containerID="0d86cefa9029acf6e8e9bd6187503fae33d28e934bda39e45ef313089b1323aa"
Oct 29 23:30:10.252547 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:10.252525    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl_odh-ai-gateway-infra(4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec"
Oct 29 23:30:19.011801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:19.011767    6203 scope.go:122] "RemoveContainer" containerID="c4704d052b931d21f8afee290c772d2ff6d1f93f3eb326ce30bda60db4bdc40c"
Oct 29 23:30:19.980902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:19.980180    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-qpzfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:30:19.980902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:19.980284    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:20.011981 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:20.011948    6203 scope.go:122] "RemoveContainer" containerID="b7afc30134ece2ed53a257a1e8428e8649703900206e2afc8e6172ff40e5f44d"
Oct 29 23:30:20.012408 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:20.012070    6203 scope.go:122] "RemoveContainer" containerID="f3bee1555f3489ba3cdf74ae671e1841aa06672e05445c3c89a11ce54297b333"
Oct 29 23:30:20.414889 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:20.414843    6203 generic.go:356] "Generic (PLEG): container finished" podID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerID="5e8f6e044c08cbfaee3d2b7c6a859ea95a06eb276d1a0d4f5fe234f3a190125b" exitCode=60
Oct 29 23:30:20.415063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:20.414912    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" event={"ID":"7bc970d5-8060-4e34-bd76-ea0d0edf9b84","Type":"ContainerDied","Data":"5e8f6e044c08cbfaee3d2b7c6a859ea95a06eb276d1a0d4f5fe234f3a190125b"}
Oct 29 23:30:20.415063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:20.415007    6203 scope.go:122] "RemoveContainer" containerID="c4704d052b931d21f8afee290c772d2ff6d1f93f3eb326ce30bda60db4bdc40c"
Oct 29 23:30:20.415402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:20.415372    6203 scope.go:122] "RemoveContainer" containerID="5e8f6e044c08cbfaee3d2b7c6a859ea95a06eb276d1a0d4f5fe234f3a190125b"
Oct 29 23:30:20.415726 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:20.415692    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd_odh-ai-gateway-infra(7bc970d5-8060-4e34-bd76-ea0d0edf9b84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84"
Oct 29 23:30:20.417015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:20.416993    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" event={"ID":"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89","Type":"ContainerStarted","Data":"40afbab62799b7167b832efdbb4a8b1cda19c82eb35f0580b11df59f1b456d2d"}
Oct 29 23:30:21.011324 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.011293    6203 scope.go:122] "RemoveContainer" containerID="bd7d73b7dc6400abb154a56298b3fb491d41cd548302c3195d7801dc17a461c3"
Oct 29 23:30:21.429581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.429543    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" event={"ID":"18c1ce70-9018-4a6a-b823-3b497c0b6b47","Type":"ContainerStarted","Data":"b01b0e873c1377795230c462947cca83b818e420b43ee7ccff780502ad6f1dde"}
Oct 29 23:30:21.433171 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.433144    6203 generic.go:356] "Generic (PLEG): container finished" podID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerID="d1b2028e74693234168a4c55d4b61af66108407070087c1c11d59fd795eca9c0" exitCode=60
Oct 29 23:30:21.433313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.433214    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" event={"ID":"a80b5e9e-d740-4a82-9c69-78ef95ba34c0","Type":"ContainerDied","Data":"d1b2028e74693234168a4c55d4b61af66108407070087c1c11d59fd795eca9c0"}
Oct 29 23:30:21.433313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.433266    6203 scope.go:122] "RemoveContainer" containerID="f3bee1555f3489ba3cdf74ae671e1841aa06672e05445c3c89a11ce54297b333"
Oct 29 23:30:21.433639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.433619    6203 scope.go:122] "RemoveContainer" containerID="d1b2028e74693234168a4c55d4b61af66108407070087c1c11d59fd795eca9c0"
Oct 29 23:30:21.433951 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:21.433930    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888610-9qntv_odh-ai-gateway-infra(a80b5e9e-d740-4a82-9c69-78ef95ba34c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0"
Oct 29 23:30:21.435230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.435207    6203 generic.go:356] "Generic (PLEG): container finished" podID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerID="40afbab62799b7167b832efdbb4a8b1cda19c82eb35f0580b11df59f1b456d2d" exitCode=60
Oct 29 23:30:21.435318 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.435274    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" event={"ID":"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89","Type":"ContainerDied","Data":"40afbab62799b7167b832efdbb4a8b1cda19c82eb35f0580b11df59f1b456d2d"}
Oct 29 23:30:21.435568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.435551    6203 scope.go:122] "RemoveContainer" containerID="40afbab62799b7167b832efdbb4a8b1cda19c82eb35f0580b11df59f1b456d2d"
Oct 29 23:30:21.435794 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:21.435778    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb_odh-ai-gateway-infra(5a1f8c79-cf53-4cb9-8dad-c65fc4423c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89"
Oct 29 23:30:21.445245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.445227    6203 scope.go:122] "RemoveContainer" containerID="b7afc30134ece2ed53a257a1e8428e8649703900206e2afc8e6172ff40e5f44d"
Oct 29 23:30:21.473524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.473497    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"]
Oct 29 23:30:21.519077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.518987    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"]
Oct 29 23:30:21.632283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.632254    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:21.747688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.747660    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-kube-api-access-t2lkc\" (UniqueName: \"kubernetes.io/projected/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-kube-api-access-t2lkc\") pod \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") "
Oct 29 23:30:21.747916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.747802    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-maas-api-service-ca\") pod \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\" (UID: \"7bc970d5-8060-4e34-bd76-ea0d0edf9b84\") "
Oct 29 23:30:21.748172 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.748148    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-maas-api-service-ca" pod "7bc970d5-8060-4e34-bd76-ea0d0edf9b84" (UID: "7bc970d5-8060-4e34-bd76-ea0d0edf9b84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:21.748292 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.748277    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-maas-api-service-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:21.749686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.749665    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-kube-api-access-t2lkc" pod "7bc970d5-8060-4e34-bd76-ea0d0edf9b84" (UID: "7bc970d5-8060-4e34-bd76-ea0d0edf9b84"). InnerVolumeSpecName "kube-api-access-t2lkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:21.849432 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:21.849398    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2lkc\" (UniqueName: \"kubernetes.io/projected/7bc970d5-8060-4e34-bd76-ea0d0edf9b84-kube-api-access-t2lkc\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:22.015954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.015928    6203 scope.go:122] "RemoveContainer" containerID="0d86cefa9029acf6e8e9bd6187503fae33d28e934bda39e45ef313089b1323aa"
Oct 29 23:30:22.444538 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.444499    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd" event={"ID":"7bc970d5-8060-4e34-bd76-ea0d0edf9b84","Type":"ContainerDied","Data":"c2c055f969d80726c981db8a40eb6897dd12480d8b7ab025c67ef197a4f984d0"}
Oct 29 23:30:22.445863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.444573    6203 scope.go:122] "RemoveContainer" containerID="5e8f6e044c08cbfaee3d2b7c6a859ea95a06eb276d1a0d4f5fe234f3a190125b"
Oct 29 23:30:22.445863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.444510    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"
Oct 29 23:30:22.454046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.454019    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerStarted","Data":"9f4f793bc515b830eedd600a41cdb3a24d432b4aa513c1e35651daa79e1832e3"}
Oct 29 23:30:22.456407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.456382    6203 generic.go:356] "Generic (PLEG): container finished" podID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerID="b01b0e873c1377795230c462947cca83b818e420b43ee7ccff780502ad6f1dde" exitCode=60
Oct 29 23:30:22.456510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.456431    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" event={"ID":"18c1ce70-9018-4a6a-b823-3b497c0b6b47","Type":"ContainerDied","Data":"b01b0e873c1377795230c462947cca83b818e420b43ee7ccff780502ad6f1dde"}
Oct 29 23:30:22.456833 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.456813    6203 scope.go:122] "RemoveContainer" containerID="b01b0e873c1377795230c462947cca83b818e420b43ee7ccff780502ad6f1dde"
Oct 29 23:30:22.457618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.457598    6203 scope.go:122] "RemoveContainer" containerID="bd7d73b7dc6400abb154a56298b3fb491d41cd548302c3195d7801dc17a461c3"
Oct 29 23:30:22.459163 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:22.459138    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k_odh-ai-gateway-infra(18c1ce70-9018-4a6a-b823-3b497c0b6b47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47"
Oct 29 23:30:22.492187 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.492148    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"]
Oct 29 23:30:22.642534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.642498    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"]
Oct 29 23:30:22.668922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.667923    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:22.668922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.668507    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"]
Oct 29 23:30:22.689621 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.689580    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-92f6b3-29888610-bpftd"]
Oct 29 23:30:22.706248 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.706187    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"]
Oct 29 23:30:22.707138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707110    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:22.707138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707135    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707155    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707163    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707185    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707192    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707206    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707213    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707230    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.707344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707237    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.707710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707408    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:22.707710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707421    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.707710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707432    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:22.707710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.707441    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:22.712333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.712308    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.754936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.751384    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"]
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.760545    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-kube-api-access-9qh8t\" (UniqueName: \"kubernetes.io/projected/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-kube-api-access-9qh8t\") pod \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") "
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.760605    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-maas-api-service-ca\") pod \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\" (UID: \"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89\") "
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761020    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761079    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761111    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761203    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761227    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761284    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761321    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vfq\" (UniqueName: \"kubernetes.io/projected/bf9e8ad1-a722-4619-94b6-32d6e97002fc-kube-api-access-57vfq\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761360    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.761838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.761406    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.766313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.766273    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-maas-api-service-ca" pod "5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" (UID: "5a1f8c79-cf53-4cb9-8dad-c65fc4423c89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:22.768593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.768566    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-kube-api-access-9qh8t" pod "5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" (UID: "5a1f8c79-cf53-4cb9-8dad-c65fc4423c89"). InnerVolumeSpecName "kube-api-access-9qh8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:22.839731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.839687    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:22.878699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.878649    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.878933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.878798    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.878933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.878823    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.878933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.878929    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.878948    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.878987    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879024    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vfq\" (UniqueName: \"kubernetes.io/projected/bf9e8ad1-a722-4619-94b6-32d6e97002fc-kube-api-access-57vfq\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879062    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879095    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879179    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qh8t\" (UniqueName: \"kubernetes.io/projected/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-kube-api-access-9qh8t\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:22.879339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879190    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89-maas-api-service-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:22.879339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879216    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879434    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.879921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.879870    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.880068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.880038    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.880182 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.880159    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.885273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.885206    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.885501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.885468    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.906464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.906438    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vfq\" (UniqueName: \"kubernetes.io/projected/bf9e8ad1-a722-4619-94b6-32d6e97002fc-kube-api-access-57vfq\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.906813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.906760    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9e8ad1-a722-4619-94b6-32d6e97002fc-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-vqnqf\" (UID: \"bf9e8ad1-a722-4619-94b6-32d6e97002fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:22.980276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.980240    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-maas-api-service-ca\") pod \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") "
Oct 29 23:30:22.980448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.980413    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-kube-api-access-h2vz2\" (UniqueName: \"kubernetes.io/projected/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-kube-api-access-h2vz2\") pod \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\" (UID: \"a80b5e9e-d740-4a82-9c69-78ef95ba34c0\") "
Oct 29 23:30:22.980602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.980576    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-maas-api-service-ca" pod "a80b5e9e-d740-4a82-9c69-78ef95ba34c0" (UID: "a80b5e9e-d740-4a82-9c69-78ef95ba34c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:22.980824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.980807    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-maas-api-service-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:22.982442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:22.982420    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-kube-api-access-h2vz2" pod "a80b5e9e-d740-4a82-9c69-78ef95ba34c0" (UID: "a80b5e9e-d740-4a82-9c69-78ef95ba34c0"). InnerVolumeSpecName "kube-api-access-h2vz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:23.028357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.028321    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:23.082063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.082035    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2vz2\" (UniqueName: \"kubernetes.io/projected/a80b5e9e-d740-4a82-9c69-78ef95ba34c0-kube-api-access-h2vz2\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:23.352667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.352627    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"]
Oct 29 23:30:23.352954 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:30:23.352928    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9e8ad1_a722_4619_94b6_32d6e97002fc.slice/crio-580d2174cb5ad7eb9530d8156ef47dc183e5bce6453518b576f1b4734ecb8f2a WatchSource:0}: Error finding container 580d2174cb5ad7eb9530d8156ef47dc183e5bce6453518b576f1b4734ecb8f2a: Status 404 returned error can't find the container with id 580d2174cb5ad7eb9530d8156ef47dc183e5bce6453518b576f1b4734ecb8f2a
Oct 29 23:30:23.355562 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.355529    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:23.355624 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.355599    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:23.355664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.355629    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:23.468401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.468370    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"
Oct 29 23:30:23.468401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.468389    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv" event={"ID":"a80b5e9e-d740-4a82-9c69-78ef95ba34c0","Type":"ContainerDied","Data":"9c78ae7edd5494b89fac5464699f636d383ba26ecd3d3e1ad5319d56eb5aed49"}
Oct 29 23:30:23.468961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.468455    6203 scope.go:122] "RemoveContainer" containerID="d1b2028e74693234168a4c55d4b61af66108407070087c1c11d59fd795eca9c0"
Oct 29 23:30:23.470678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.470656    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"
Oct 29 23:30:23.470678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.470664    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb" event={"ID":"5a1f8c79-cf53-4cb9-8dad-c65fc4423c89","Type":"ContainerDied","Data":"3eefdab6ccad8ba923fccbdca210ecb17c0bd31536473b773c45a82c26219e64"}
Oct 29 23:30:23.473281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.473145    6203 generic.go:356] "Generic (PLEG): container finished" podID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerID="9f4f793bc515b830eedd600a41cdb3a24d432b4aa513c1e35651daa79e1832e3" exitCode=60
Oct 29 23:30:23.474256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.473229    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerDied","Data":"9f4f793bc515b830eedd600a41cdb3a24d432b4aa513c1e35651daa79e1832e3"}
Oct 29 23:30:23.474731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.474712    6203 scope.go:122] "RemoveContainer" containerID="9f4f793bc515b830eedd600a41cdb3a24d432b4aa513c1e35651daa79e1832e3"
Oct 29 23:30:23.475129 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:23.475087    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl_odh-ai-gateway-infra(4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec"
Oct 29 23:30:23.478185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.478156    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" event={"ID":"bf9e8ad1-a722-4619-94b6-32d6e97002fc","Type":"ContainerStarted","Data":"3df8efb581c9d668f44c0c210d394edb917c7e50fe9cef486bd7e3116261c929"}
Oct 29 23:30:23.478306 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.478207    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" event={"ID":"bf9e8ad1-a722-4619-94b6-32d6e97002fc","Type":"ContainerStarted","Data":"580d2174cb5ad7eb9530d8156ef47dc183e5bce6453518b576f1b4734ecb8f2a"}
Oct 29 23:30:23.482532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.482514    6203 scope.go:122] "RemoveContainer" containerID="40afbab62799b7167b832efdbb4a8b1cda19c82eb35f0580b11df59f1b456d2d"
Oct 29 23:30:23.494108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.493343    6203 scope.go:122] "RemoveContainer" containerID="0d86cefa9029acf6e8e9bd6187503fae33d28e934bda39e45ef313089b1323aa"
Oct 29 23:30:23.530339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.530306    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"]
Oct 29 23:30:23.534742 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.534214    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888610-9qntv"]
Oct 29 23:30:23.579258 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.579228    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"]
Oct 29 23:30:23.656075 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.656047    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:23.698643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.698594    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"]
Oct 29 23:30:23.702251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.702224    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ae1784-29888610-rrrdb"]
Oct 29 23:30:23.753391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.753334    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podStartSLOduration=1.753320513 podStartE2EDuration="1.753320513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:23.751033525 +0000 UTC m=+1001.855293936" watchObservedRunningTime="2026-10-29 23:30:23.753320513 +0000 UTC m=+1001.857580888"
Oct 29 23:30:23.806118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.806088    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c1ce70-9018-4a6a-b823-3b497c0b6b47-kube-api-access-k5c6l\" (UniqueName: \"kubernetes.io/projected/18c1ce70-9018-4a6a-b823-3b497c0b6b47-kube-api-access-k5c6l\") pod \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") "
Oct 29 23:30:23.806118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.806132    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c1ce70-9018-4a6a-b823-3b497c0b6b47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c1ce70-9018-4a6a-b823-3b497c0b6b47-maas-api-service-ca\") pod \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\" (UID: \"18c1ce70-9018-4a6a-b823-3b497c0b6b47\") "
Oct 29 23:30:23.806911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.806869    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c1ce70-9018-4a6a-b823-3b497c0b6b47-maas-api-service-ca" pod "18c1ce70-9018-4a6a-b823-3b497c0b6b47" (UID: "18c1ce70-9018-4a6a-b823-3b497c0b6b47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:23.809047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.809025    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c1ce70-9018-4a6a-b823-3b497c0b6b47-kube-api-access-k5c6l" pod "18c1ce70-9018-4a6a-b823-3b497c0b6b47" (UID: "18c1ce70-9018-4a6a-b823-3b497c0b6b47"). InnerVolumeSpecName "kube-api-access-k5c6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:23.913704 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.913592    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18c1ce70-9018-4a6a-b823-3b497c0b6b47-maas-api-service-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:23.913704 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:23.913635    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5c6l\" (UniqueName: \"kubernetes.io/projected/18c1ce70-9018-4a6a-b823-3b497c0b6b47-kube-api-access-k5c6l\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:24.019590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.019510    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" path="/var/lib/kubelet/pods/5a1f8c79-cf53-4cb9-8dad-c65fc4423c89/volumes"
Oct 29 23:30:24.019972 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.019955    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" path="/var/lib/kubelet/pods/7bc970d5-8060-4e34-bd76-ea0d0edf9b84/volumes"
Oct 29 23:30:24.020401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.020386    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" path="/var/lib/kubelet/pods/a80b5e9e-d740-4a82-9c69-78ef95ba34c0/volumes"
Oct 29 23:30:24.039912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.032182    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:24.055912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.055381    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:24.055912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.055442    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:24.490077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.490043    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"
Oct 29 23:30:24.490433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.490052    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k" event={"ID":"18c1ce70-9018-4a6a-b823-3b497c0b6b47","Type":"ContainerDied","Data":"5acd1d755c94f5d85e0532bcf6bc8f9b04ecebc7081cbcffb79910e7f0b5ab25"}
Oct 29 23:30:24.490433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.490228    6203 scope.go:122] "RemoveContainer" containerID="b01b0e873c1377795230c462947cca83b818e420b43ee7ccff780502ad6f1dde"
Oct 29 23:30:24.567203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567164    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"]
Oct 29 23:30:24.567814 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567792    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.567814 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567815    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567838    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567845    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567856    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567864    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567912    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567920    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567929    6203 cpu_manager.go:414] "containerMap: removing container" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567936    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567946    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567953    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567969    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:24.568026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.567975    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:24.568839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.568360    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.568383    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.568394    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.568410    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" containerName="cleanup"
Oct 29 23:30:24.568839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.568422    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:24.568839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.568437    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1f8c79-cf53-4cb9-8dad-c65fc4423c89" containerName="cleanup"
Oct 29 23:30:24.568839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.568447    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc970d5-8060-4e34-bd76-ea0d0edf9b84" containerName="cleanup"
Oct 29 23:30:24.573589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.573567    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.596724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.595576    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"]
Oct 29 23:30:24.661145 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.660955    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"]
Oct 29 23:30:24.669321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.667055    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-aa7e96-29888610-whg7k"]
Oct 29 23:30:24.723717 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.723688    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152dd6e0-c036-4a0b-a15f-a5cac127517e-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.723923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.723905    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.724039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.724028    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.724150 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.724134    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.724257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.724246    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.724437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.724418    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.724496 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.724449    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.724496 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.724475    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9k6d\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-kube-api-access-d9k6d\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.724575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.724507    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.726772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.726754    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.825395    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-kube-api-access-wj2dl\" (UniqueName: \"kubernetes.io/projected/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-kube-api-access-wj2dl\") pod \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") "
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.825602    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-maas-api-service-ca\") pod \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\" (UID: \"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec\") "
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826018    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826156    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152dd6e0-c036-4a0b-a15f-a5cac127517e-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826208    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826222    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-maas-api-service-ca" pod "4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" (UID: "4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826305    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826376    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826401    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826534    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826567    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826607    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9k6d\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-kube-api-access-d9k6d\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826707    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-maas-api-service-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.826769    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.827725    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152dd6e0-c036-4a0b-a15f-a5cac127517e-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.827766    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.850950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.828110    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.853903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.853024    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-kube-api-access-wj2dl" pod "4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" (UID: "4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec"). InnerVolumeSpecName "kube-api-access-wj2dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:30:24.853903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.853346    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.857427 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.857398    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9k6d\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-kube-api-access-d9k6d\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.860922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.860699    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.860922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.860869    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.863187 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.863150    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.898497 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.898456    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:24.943913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:24.928510    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj2dl\" (UniqueName: \"kubernetes.io/projected/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec-kube-api-access-wj2dl\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:30:25.057904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.057817    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:25.058064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.057911    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:25.254977 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.254934    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"]
Oct 29 23:30:25.261627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.261592    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:25.261719 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.261681    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:25.261719 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.261715    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:25.499509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.499469    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" event={"ID":"152dd6e0-c036-4a0b-a15f-a5cac127517e","Type":"ContainerStarted","Data":"113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13"}
Oct 29 23:30:25.500009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.499531    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" event={"ID":"152dd6e0-c036-4a0b-a15f-a5cac127517e","Type":"ContainerStarted","Data":"a76944e46ff357042a71585964ff17a010e61c202a1a6f2fa5c92db7d4a60a47"}
Oct 29 23:30:25.501018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.500989    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"
Oct 29 23:30:25.501141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.500990    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl" event={"ID":"4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec","Type":"ContainerDied","Data":"24a162dffec81a646a53d4ec5a8ecdcf0202cf8758d4203dc6534d828bbcd6b7"}
Oct 29 23:30:25.501141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.501128    6203 scope.go:122] "RemoveContainer" containerID="9f4f793bc515b830eedd600a41cdb3a24d432b4aa513c1e35651daa79e1832e3"
Oct 29 23:30:25.559771 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.559708    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podStartSLOduration=1.559690096 podStartE2EDuration="1.559690096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:25.559321094 +0000 UTC m=+1003.663581477" watchObservedRunningTime="2026-10-29 23:30:25.559690096 +0000 UTC m=+1003.663950484"
Oct 29 23:30:25.609352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.609264    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"]
Oct 29 23:30:25.615154 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.615123    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ae1364-29888610-f57wl"]
Oct 29 23:30:25.898701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:25.898610    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:26.028902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:26.027304    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c1ce70-9018-4a6a-b823-3b497c0b6b47" path="/var/lib/kubelet/pods/18c1ce70-9018-4a6a-b823-3b497c0b6b47/volumes"
Oct 29 23:30:26.028902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:26.028054    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" path="/var/lib/kubelet/pods/4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec/volumes"
Oct 29 23:30:26.051927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:26.038038    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:26.051927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:26.038110    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:26.900521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:26.900478    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 23:30:26.900939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:26.900542    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 23:30:27.032322 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:27.032282    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:27.032614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:27.032588    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:27.898925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:27.898890    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 23:30:27.899135 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:27.898948    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 23:30:28.048902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.048565    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:28.048902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.048628    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:28.483172 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483134    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"]
Oct 29 23:30:28.483715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483699    6203 cpu_manager.go:414] "containerMap: removing container" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483717    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483738    6203 cpu_manager.go:414] "containerMap: removing container" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483744    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483755    6203 cpu_manager.go:414] "containerMap: removing container" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483761    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483860    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483869    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80b5e9e-d740-4a82-9c69-78ef95ba34c0" containerName="cleanup"
Oct 29 23:30:28.483943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483891    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.483943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.483897    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4e7ebe-7bc7-4f54-b7ab-f66a35c6b5ec" containerName="cleanup"
Oct 29 23:30:28.488629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.488611    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.506648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.506625    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-model-cache\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.506755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.506656    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-home\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.506755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.506681    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-dshm\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.506755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.506730    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgrl8\" (UniqueName: \"kubernetes.io/projected/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-kube-api-access-lgrl8\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.506849 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.506802    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tls-certs\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.506849 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.506829    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tmp-dir\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.524591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.523485    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"]
Oct 29 23:30:28.547933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.528201    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b978a0\"/\"distinct-w0-b978a0-kserve-self-signed-certs\""
Oct 29 23:30:28.607641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.607610    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tls-certs\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.607824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.607657    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tmp-dir\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.607824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.607715    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-model-cache\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.607824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.607738    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-home\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.607824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.607776    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-dshm\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.608033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.607857    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgrl8\" (UniqueName: \"kubernetes.io/projected/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-kube-api-access-lgrl8\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.608185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.608164    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tmp-dir\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.608269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.608250    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-home\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.608520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.608504    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-model-cache\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.610932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.610794    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-dshm\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.610932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.610868    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tls-certs\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.687005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.685529    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgrl8\" (UniqueName: \"kubernetes.io/projected/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-kube-api-access-lgrl8\") pod \"distinct-w0-b978a0-kserve-64b56ff8fd-9h92m\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") " pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.801695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.801139    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:28.899628 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.899270    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 23:30:28.899628 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:28.899321    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 23:30:29.080966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.080073    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:29.080966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.080151    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:29.184203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.184042    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"]
Oct 29 23:30:29.188682 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:30:29.188647    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b8b6c76_2dfb_4c01_a46d_9b88a001d994.slice/crio-19c38d360943ceff3993e44dd98d601951c7b49daa07592bb7b2a6b6c480e511 WatchSource:0}: Error finding container 19c38d360943ceff3993e44dd98d601951c7b49daa07592bb7b2a6b6c480e511: Status 404 returned error can't find the container with id 19c38d360943ceff3993e44dd98d601951c7b49daa07592bb7b2a6b6c480e511
Oct 29 23:30:29.600927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.600861    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" event={"ID":"9b8b6c76-2dfb-4c01-a46d-9b88a001d994","Type":"ContainerStarted","Data":"f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570"}
Oct 29 23:30:29.601104 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.600939    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" event={"ID":"9b8b6c76-2dfb-4c01-a46d-9b88a001d994","Type":"ContainerStarted","Data":"19c38d360943ceff3993e44dd98d601951c7b49daa07592bb7b2a6b6c480e511"}
Oct 29 23:30:29.657529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.657451    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podStartSLOduration=1.657427061 podStartE2EDuration="1.657427061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:29.653127157 +0000 UTC m=+1007.757387546" watchObservedRunningTime="2026-10-29 23:30:29.657427061 +0000 UTC m=+1007.761687448"
Oct 29 23:30:29.809026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.808986    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:29.903027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.899232    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 23:30:29.903027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:29.899306    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 23:30:30.033901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:30.033010    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:30.033901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:30.033073    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:30.899118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:30.899076    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 23:30:30.899553 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:30.899140    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 23:30:31.043009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:31.042971    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 23:30:31.043259 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:31.043227    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 23:30:31.899103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:31.899057    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 29 23:30:31.899307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:31.899125    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 29 23:30:32.362915 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:32.362866    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:32.363506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:32.363479    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:32.364501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:32.364477    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf"
Oct 29 23:30:33.074364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:33.074323    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:33.074912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:33.074869    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:33.075870 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:33.075839    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:30:38.802552 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:38.802519    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:38.818232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:38.818203    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:39.680440 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:39.680411    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:30:39.924474 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:30:39.924433    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:30:53.573186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.573150    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"]
Oct 29 23:30:53.579381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.579356    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.604902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.604854    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"]
Oct 29 23:30:53.678269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678232    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678273    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnf5\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-kube-api-access-njnf5\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678296    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678427    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678461    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678485    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678504    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678527    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.678705 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.678676    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780148 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780108    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780152    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnf5\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-kube-api-access-njnf5\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780189    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780242    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780297    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780338    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780369    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780406    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780456    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780577    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780799    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.780925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780866    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.781013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780918    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.781013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.780969    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.782621 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.782593    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.782873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.782853    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.806405 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.806380    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.806535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.806448    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnf5\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-kube-api-access-njnf5\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.893834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.893758    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:53.997574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:53.997535    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"]
Oct 29 23:30:54.004021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.003991    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.029012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.028985    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"]
Oct 29 23:30:54.038464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.038432    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"]
Oct 29 23:30:54.042480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.042462    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.076513 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.076478    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"]
Oct 29 23:30:54.084316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084281    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.084459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084401    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.084520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084466    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.084573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084554    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.084614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084586    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0d2e4a-833c-40ef-9d12-8318ec337964-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.084661 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084645    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.084803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084787    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.084893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084827    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.084893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084845    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.085004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.084974    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjz7\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-kube-api-access-nwjz7\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.085056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085009    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.085056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085040    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.085167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085145    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntwdl\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-kube-api-access-ntwdl\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.085213 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085197    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.085287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085275    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.085328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085300    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.085402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085384    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.085442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.085427    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.186425 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186390    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.186597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186443    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.186597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186460    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.186597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186499    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjz7\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-kube-api-access-nwjz7\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.186597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186519    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.186597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186536    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.186892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186707    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntwdl\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-kube-api-access-ntwdl\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.186892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186776    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.186892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186838    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.186892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186889    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186868    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186947    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.186952    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187026    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187083    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187158    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187176    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187192    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187279    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187305    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0d2e4a-833c-40ef-9d12-8318ec337964-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187344    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187417    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187568    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187675    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.187839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187727    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-data\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.187839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187829    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-credential-socket\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.188059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.187973    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-certs\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.188151 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.188129    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0d2e4a-833c-40ef-9d12-8318ec337964-istiod-ca-cert\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.189066 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.189040    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.189235 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.189218    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-envoy\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.189709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.189686    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.189795 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.189742    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-podinfo\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.212805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.212777    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"]
Oct 29 23:30:54.213182 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:30:54.213165    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6afa2e8_8160_4df4_ade8_9c1b4eec46ba.slice/crio-af80707ec924fbc08ae3e39d96738ba56fe0ec7b95c614e0099d8559b5406d3a WatchSource:0}: Error finding container af80707ec924fbc08ae3e39d96738ba56fe0ec7b95c614e0099d8559b5406d3a: Status 404 returned error can't find the container with id af80707ec924fbc08ae3e39d96738ba56fe0ec7b95c614e0099d8559b5406d3a
Oct 29 23:30:54.215742 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.215709    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.215808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.215784    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.215848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.215837    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.220581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.220546    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjz7\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-kube-api-access-nwjz7\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.220683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.220589    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntwdl\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-kube-api-access-ntwdl\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.220683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.220621    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.222295 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.222272    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-token\") pod \"e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") " pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.322093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.322060    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:54.363210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.363174    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:54.641602 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:30:54.641552    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0d2e4a_833c_40ef_9d12_8318ec337964.slice/crio-0103c319bc7c3eef5ab18445e4fda8b268fcabd856ad2e52781afaba17051931 WatchSource:0}: Error finding container 0103c319bc7c3eef5ab18445e4fda8b268fcabd856ad2e52781afaba17051931: Status 404 returned error can't find the container with id 0103c319bc7c3eef5ab18445e4fda8b268fcabd856ad2e52781afaba17051931
Oct 29 23:30:54.642007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.641710    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"]
Oct 29 23:30:54.644523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.644490    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.644633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.644553    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.644633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.644582    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.692907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.692866    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"]
Oct 29 23:30:54.693438 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:30:54.693412    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad364fdf_8cdf_419b_b95d_35d20e57da3a.slice/crio-b586585ad109fd8ec998cd74ea7c6b947fbee8766a6b55a5a14ae3d42a381351 WatchSource:0}: Error finding container b586585ad109fd8ec998cd74ea7c6b947fbee8766a6b55a5a14ae3d42a381351: Status 404 returned error can't find the container with id b586585ad109fd8ec998cd74ea7c6b947fbee8766a6b55a5a14ae3d42a381351
Oct 29 23:30:54.695998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.695959    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.696087 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.696030    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.696087 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.696059    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:30:54.786233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.786204    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" event={"ID":"ad364fdf-8cdf-419b-b95d-35d20e57da3a","Type":"ContainerStarted","Data":"b586585ad109fd8ec998cd74ea7c6b947fbee8766a6b55a5a14ae3d42a381351"}
Oct 29 23:30:54.788740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.788314    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" event={"ID":"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba","Type":"ContainerStarted","Data":"b2a613f843cd578eed854a2977286a5c4654b4bbef9846d487d5d4f255149daf"}
Oct 29 23:30:54.788956 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.788933    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" event={"ID":"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba","Type":"ContainerStarted","Data":"af80707ec924fbc08ae3e39d96738ba56fe0ec7b95c614e0099d8559b5406d3a"}
Oct 29 23:30:54.790456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.790395    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" event={"ID":"ec0d2e4a-833c-40ef-9d12-8318ec337964","Type":"ContainerStarted","Data":"bd048738381018deac50f49006c4272272668e3c83a95c5574cdbe0b26b58f25"}
Oct 29 23:30:54.790569 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.790475    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" event={"ID":"ec0d2e4a-833c-40ef-9d12-8318ec337964","Type":"ContainerStarted","Data":"0103c319bc7c3eef5ab18445e4fda8b268fcabd856ad2e52781afaba17051931"}
Oct 29 23:30:54.839462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.839399    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podStartSLOduration=1.839380317 podStartE2EDuration="1.839380317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:54.838539844 +0000 UTC m=+1032.942800234" watchObservedRunningTime="2026-10-29 23:30:54.839380317 +0000 UTC m=+1032.943640706"
Oct 29 23:30:54.893919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.893846    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podStartSLOduration=1.893832454 podStartE2EDuration="1.893832454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:54.890534709 +0000 UTC m=+1032.994795096" watchObservedRunningTime="2026-10-29 23:30:54.893832454 +0000 UTC m=+1032.998092839"
Oct 29 23:30:54.894808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:54.894783    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:30:55.322983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:55.322944    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:30:55.324404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:55.324375    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 23:30:55.324553 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:55.324432    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 23:30:55.809907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:55.807619    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" event={"ID":"ad364fdf-8cdf-419b-b95d-35d20e57da3a","Type":"ContainerStarted","Data":"7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e"}
Oct 29 23:30:55.905904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:55.902037    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 23:30:55.905904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:55.902096    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 23:30:55.905904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:55.905079    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podStartSLOduration=1.905058176 podStartE2EDuration="1.905058176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:30:55.900102017 +0000 UTC m=+1034.004362384" watchObservedRunningTime="2026-10-29 23:30:55.905058176 +0000 UTC m=+1034.009318569"
Oct 29 23:30:56.323511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:56.323475    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 23:30:56.323695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:56.323533    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 23:30:56.378734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:56.375976    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:30:56.388911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:56.388199    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 23:30:56.388911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:56.388262    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 23:30:56.907901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:56.907471    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 23:30:56.907901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:56.907532    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 23:30:57.323458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:57.323422    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 23:30:57.323616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:57.323475    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 23:30:57.367902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:57.365311    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 23:30:57.367902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:57.365379    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 23:30:57.906902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:57.906575    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 23:30:57.906902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:57.906636    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 23:30:58.325902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:58.325074    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 23:30:58.325902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:58.325141    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 23:30:58.374903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:58.369729    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 23:30:58.374903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:58.369795    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 23:30:58.906513 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:58.906476    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 23:30:58.906790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:58.906754    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 23:30:59.227902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.226944    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:30:59.338522 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.338482    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 23:30:59.339500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.339465    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 23:30:59.377903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.364722    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 23:30:59.377903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.364780    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 23:30:59.686196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.683976    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:30:59.894900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.894273    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 23:30:59.894900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.894336    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 23:30:59.944975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:30:59.944851    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:31:00.050902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.050109    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:00.093903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.088035    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:00.093903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.088396    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:00.323043 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.322943    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 23:31:00.323043 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.323004    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 23:31:00.381904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.381016    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 23:31:00.381904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.381093    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 23:31:00.896076 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.896034    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 23:31:00.896314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:00.896112    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 23:31:01.323079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:01.323040    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 29 23:31:01.323250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:01.323104    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 29 23:31:01.364166 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:01.364125    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 23:31:01.364327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:01.364208    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 23:31:02.275105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.275070    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:31:02.276432 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.276397    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:31:02.279331 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.279309    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:31:02.515919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.515181    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:31:02.517174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.517151    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:31:02.519183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.519160    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:31:02.820831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.820796    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:31:02.821214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.821194    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:31:02.822175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:02.822153    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:31:08.733382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.733350    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"]
Oct 29 23:31:08.737840 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.737818    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.765265 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.765231    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"]
Oct 29 23:31:08.778477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.778451    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"]
Oct 29 23:31:08.782772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.782752    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.808362 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.808330    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"]
Oct 29 23:31:08.843245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843213    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.843245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843248    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.843493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843267    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.843493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843411    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.843493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843447    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.843650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843549    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.843650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843620    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.843781 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843759    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7lf\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-kube-api-access-5h7lf\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.843818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843801    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.843908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843889    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7394c95c-d268-48c5-9c70-f065b1a8a536-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.843977 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843935    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ee719e-10b5-4394-a7d0-6820eda97835-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.844028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.843973    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.844028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.844017    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ee719e-10b5-4394-a7d0-6820eda97835-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.844123 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.844051    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.844123 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.844074    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.844219 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.844176    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.844263 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.844245    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.844305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.844289    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpn7m\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-kube-api-access-gpn7m\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.945310 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945268    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.945488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945318    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ee719e-10b5-4394-a7d0-6820eda97835-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.945488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945340    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.945488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945363    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.945488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945415    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.945488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945472    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.945758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945532    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpn7m\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-kube-api-access-gpn7m\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.945758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945577    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.945758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945608    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.945758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945651    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.945758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945700    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.945758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945733    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.946111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945782    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.946111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945847    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.946111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945861    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.946111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945866    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.946111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.945958    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.946111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946006    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-certs\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.946444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946274    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.946444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946306    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.946444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946315    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7lf\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-kube-api-access-5h7lf\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.946444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946345    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-data\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.946444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946357    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.946444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946374    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-credential-socket\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.946444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946394    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7394c95c-d268-48c5-9c70-f065b1a8a536-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.946788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.946447    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ee719e-10b5-4394-a7d0-6820eda97835-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.947135 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.947111    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7394c95c-d268-48c5-9c70-f065b1a8a536-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.947232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.947150    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ee719e-10b5-4394-a7d0-6820eda97835-istiod-ca-cert\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.948470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.948449    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.948566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.948515    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ee719e-10b5-4394-a7d0-6820eda97835-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.948606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.948565    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-podinfo\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.948642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.948605    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-envoy\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.971612 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.971574    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.971746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.971699    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-token\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:08.971867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.971845    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpn7m\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-kube-api-access-gpn7m\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:08.971936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:08.971890    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7lf\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-kube-api-access-5h7lf\") pod \"e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") " pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:09.052837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.052748    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:09.100073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.100040    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:09.373945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.373916    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"]
Oct 29 23:31:09.374744 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:09.374718    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ee719e_10b5_4394_a7d0_6820eda97835.slice/crio-f4315441ab9562127ee4e87a6aa7788c33687d6094e940e005131a94be788400 WatchSource:0}: Error finding container f4315441ab9562127ee4e87a6aa7788c33687d6094e940e005131a94be788400: Status 404 returned error can't find the container with id f4315441ab9562127ee4e87a6aa7788c33687d6094e940e005131a94be788400
Oct 29 23:31:09.377667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.377630    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:09.377767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.377699    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:09.377767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.377727    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:09.434389 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:09.434360    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7394c95c_d268_48c5_9c70_f065b1a8a536.slice/crio-6c6f4db9610226070228b36feaf0c37c14d735a3749cecb3ea61ea045ae58657 WatchSource:0}: Error finding container 6c6f4db9610226070228b36feaf0c37c14d735a3749cecb3ea61ea045ae58657: Status 404 returned error can't find the container with id 6c6f4db9610226070228b36feaf0c37c14d735a3749cecb3ea61ea045ae58657
Oct 29 23:31:09.434977 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.434956    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"]
Oct 29 23:31:09.437464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.437427    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:09.437547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.437519    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:09.437581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:09.437565    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:09.962029 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:31:09.961982    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:31:10.194713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:10.187275    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" event={"ID":"7394c95c-d268-48c5-9c70-f065b1a8a536","Type":"ContainerStarted","Data":"19f8bcda38c8df72ad65b841442d0fbaec40b3b3c43a5a893d03e6d31889175b"}
Oct 29 23:31:10.194713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:10.187515    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" event={"ID":"7394c95c-d268-48c5-9c70-f065b1a8a536","Type":"ContainerStarted","Data":"6c6f4db9610226070228b36feaf0c37c14d735a3749cecb3ea61ea045ae58657"}
Oct 29 23:31:10.197092 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:10.197057    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" event={"ID":"44ee719e-10b5-4394-a7d0-6820eda97835","Type":"ContainerStarted","Data":"6d130e084323b11dc8df3a5b08d8c38a1e1d50e0a102e786bc1ce9070513ca7c"}
Oct 29 23:31:10.197224 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:10.197120    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" event={"ID":"44ee719e-10b5-4394-a7d0-6820eda97835","Type":"ContainerStarted","Data":"f4315441ab9562127ee4e87a6aa7788c33687d6094e940e005131a94be788400"}
Oct 29 23:31:10.258964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:10.241268    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podStartSLOduration=2.241251303 podStartE2EDuration="2.241251303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:10.240478169 +0000 UTC m=+1048.344738555" watchObservedRunningTime="2026-10-29 23:31:10.241251303 +0000 UTC m=+1048.345511688"
Oct 29 23:31:10.294457 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:10.294407    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podStartSLOduration=2.294391322 podStartE2EDuration="2.294391322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:10.292196848 +0000 UTC m=+1048.396457260" watchObservedRunningTime="2026-10-29 23:31:10.294391322 +0000 UTC m=+1048.398651708"
Oct 29 23:31:11.058769 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:11.058688    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:11.062896 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:11.062513    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 23:31:11.062896 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:11.062588    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 23:31:11.101042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:11.101011    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:11.102212 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:11.102188    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 23:31:11.102344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:11.102239    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 23:31:12.058361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:12.058324    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 23:31:12.058678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:12.058646    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 23:31:12.101471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:12.101344    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 23:31:12.101471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:12.101401    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 23:31:13.054256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:13.054217    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 23:31:13.054536 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:13.054509    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 23:31:13.102534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:13.102486    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 23:31:13.103970 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:13.103908    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 23:31:14.064902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:14.064725    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 23:31:14.064902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:14.064799    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 23:31:14.101331 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:14.101297    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 23:31:14.101507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:14.101352    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 23:31:15.065694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.065656    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 23:31:15.066983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.066948    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 23:31:15.091904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.085950    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"]
Oct 29 23:31:15.100813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.100784    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.107634 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.104207    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 23:31:15.107634 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.104266    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 23:31:15.121989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.121957    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"]
Oct 29 23:31:15.134174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.134143    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:31:15.134453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.134439    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:31:15.134610 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.134599    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:15.134800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.134787    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:31:15.207981 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.207946    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj866\" (UniqueName: \"kubernetes.io/projected/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-kube-api-access-rj866\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.208171 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.208050    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.208322 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.208297    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.309697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.309662    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.309970 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.309956    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj866\" (UniqueName: \"kubernetes.io/projected/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-kube-api-access-rj866\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.310103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.310092    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.310340 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.310317    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.313042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.313017    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.383095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.383010    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj866\" (UniqueName: \"kubernetes.io/projected/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-kube-api-access-rj866\") pod \"maas-api-revoke-sub-7d05e74c-62v69\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.496387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.496352    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:15.853599 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:15.853568    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"]
Oct 29 23:31:15.855193 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:15.855167    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5121fa_65a7_4ca8_afd9_7615d5e0276f.slice/crio-89def44834ad7796d0ce476b21d151953880fa8cdd84685c2283565398928aa7 WatchSource:0}: Error finding container 89def44834ad7796d0ce476b21d151953880fa8cdd84685c2283565398928aa7: Status 404 returned error can't find the container with id 89def44834ad7796d0ce476b21d151953880fa8cdd84685c2283565398928aa7
Oct 29 23:31:16.053990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:16.053952    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 29 23:31:16.054164 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:16.054008    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 29 23:31:16.101218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:16.101132    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 29 23:31:16.101218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:16.101202    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 29 23:31:16.297772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:16.297733    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69" event={"ID":"0c5121fa-65a7-4ca8-afd9-7615d5e0276f","Type":"ContainerStarted","Data":"89def44834ad7796d0ce476b21d151953880fa8cdd84685c2283565398928aa7"}
Oct 29 23:31:17.574901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:17.574160    6203 generic.go:356] "Generic (PLEG): container finished" podID="0c5121fa-65a7-4ca8-afd9-7615d5e0276f" containerID="2f8923434589808294adc9b2d29b0c0316a4519ba32b91839d683e64cffbcb44" exitCode=0
Oct 29 23:31:17.574901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:17.574448    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69" event={"ID":"0c5121fa-65a7-4ca8-afd9-7615d5e0276f","Type":"ContainerDied","Data":"2f8923434589808294adc9b2d29b0c0316a4519ba32b91839d683e64cffbcb44"}
Oct 29 23:31:18.061970 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.061923    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Oct 29 23:31:18.063982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.062268    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:31:18.064237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.064205    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:18.078134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.078091    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:18.122910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.121478    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:31:18.122910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.121562    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:18.212085 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.212026    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 29 23:31:18.235202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.235170    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:18.262932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.235444    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:18.265575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.265548    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:18.265859 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.265751    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:18.274180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.274148    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:31:18.499373 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.499328    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:18.520692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.520654    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:18.538300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.538267    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:31:18.622146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.622106    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:31:18.622864 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:18.622828    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:19.292725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.292687    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:19.348615 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.348525    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-xks-ca\") pod \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") "
Oct 29 23:31:19.348792 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.348693    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-kube-api-access-rj866\" (UniqueName: \"kubernetes.io/projected/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-kube-api-access-rj866\") pod \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") "
Oct 29 23:31:19.348952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.348926    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-openshift-ca\") pod \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\" (UID: \"0c5121fa-65a7-4ca8-afd9-7615d5e0276f\") "
Oct 29 23:31:19.349374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.349342    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-openshift-ca" pod "0c5121fa-65a7-4ca8-afd9-7615d5e0276f" (UID: "0c5121fa-65a7-4ca8-afd9-7615d5e0276f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:19.349589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.349567    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:19.351413 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.351386    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-kube-api-access-rj866" pod "0c5121fa-65a7-4ca8-afd9-7615d5e0276f" (UID: "0c5121fa-65a7-4ca8-afd9-7615d5e0276f"). InnerVolumeSpecName "kube-api-access-rj866". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:19.351526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.351409    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-xks-ca" pod "0c5121fa-65a7-4ca8-afd9-7615d5e0276f" (UID: "0c5121fa-65a7-4ca8-afd9-7615d5e0276f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:19.451154 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.451120    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:19.451154 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.451150    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj866\" (UniqueName: \"kubernetes.io/projected/0c5121fa-65a7-4ca8-afd9-7615d5e0276f-kube-api-access-rj866\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:19.701832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.701798    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"
Oct 29 23:31:19.702234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.701838    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69" event={"ID":"0c5121fa-65a7-4ca8-afd9-7615d5e0276f","Type":"ContainerDied","Data":"89def44834ad7796d0ce476b21d151953880fa8cdd84685c2283565398928aa7"}
Oct 29 23:31:19.702234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:19.701913    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89def44834ad7796d0ce476b21d151953880fa8cdd84685c2283565398928aa7"
Oct 29 23:31:27.208481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.208423    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"]
Oct 29 23:31:27.209242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.209222    6203 cpu_manager.go:414] "containerMap: removing container" podUID="0c5121fa-65a7-4ca8-afd9-7615d5e0276f" containerName="revoke-keys"
Oct 29 23:31:27.209242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.209242    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5121fa-65a7-4ca8-afd9-7615d5e0276f" containerName="revoke-keys"
Oct 29 23:31:27.209369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.209324    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5121fa-65a7-4ca8-afd9-7615d5e0276f" containerName="revoke-keys"
Oct 29 23:31:27.214085 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.214062    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.232057 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.232030    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:31:27.232211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.232120    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:31:27.235594 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.235564    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"]
Oct 29 23:31:27.244558 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.244527    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:27.246633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.246609    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:31:27.254004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.253973    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.254316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.254286    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.254495 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.254468    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlnrm\" (UniqueName: \"kubernetes.io/projected/155fe462-1192-4c66-ac31-d4ec53e5b29e-kube-api-access-dlnrm\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.356000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.355963    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlnrm\" (UniqueName: \"kubernetes.io/projected/155fe462-1192-4c66-ac31-d4ec53e5b29e-kube-api-access-dlnrm\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.356194 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.356121    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.356269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.356240    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.356893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.356849    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.358659 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.358637    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.382209 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.382180    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlnrm\" (UniqueName: \"kubernetes.io/projected/155fe462-1192-4c66-ac31-d4ec53e5b29e-kube-api-access-dlnrm\") pod \"maas-api-revoke-sub-7e28a881-xbwnn\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.530834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.530754    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:27.927910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:27.924466    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"]
Oct 29 23:31:28.768243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:28.767495    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn" event={"ID":"155fe462-1192-4c66-ac31-d4ec53e5b29e","Type":"ContainerStarted","Data":"bad5ddcbfdc45b5b001ee407e22e7fe7257698db5faacb1f611ab8931d30b026"}
Oct 29 23:31:28.768243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:28.767568    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn" event={"ID":"155fe462-1192-4c66-ac31-d4ec53e5b29e","Type":"ContainerStarted","Data":"5abb94f3f48b9aa1e17cff25b2f8392cb7a2600a34728f99b8c5802c2c6bca7f"}
Oct 29 23:31:28.816475 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:28.816424    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn" podStartSLOduration=1.8164102149999999 podStartE2EDuration="1.816410215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:28.814127245 +0000 UTC m=+1066.918387642" watchObservedRunningTime="2026-10-29 23:31:28.816410215 +0000 UTC m=+1066.920670615"
Oct 29 23:31:29.784236 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:29.784198    6203 generic.go:356] "Generic (PLEG): container finished" podID="155fe462-1192-4c66-ac31-d4ec53e5b29e" containerID="bad5ddcbfdc45b5b001ee407e22e7fe7257698db5faacb1f611ab8931d30b026" exitCode=0
Oct 29 23:31:29.784644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:29.784282    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn" event={"ID":"155fe462-1192-4c66-ac31-d4ec53e5b29e","Type":"ContainerDied","Data":"bad5ddcbfdc45b5b001ee407e22e7fe7257698db5faacb1f611ab8931d30b026"}
Oct 29 23:31:30.966965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:30.966936    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:31.028243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.028210    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-xks-ca\") pod \"155fe462-1192-4c66-ac31-d4ec53e5b29e\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") "
Oct 29 23:31:31.028433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.028307    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155fe462-1192-4c66-ac31-d4ec53e5b29e-kube-api-access-dlnrm\" (UniqueName: \"kubernetes.io/projected/155fe462-1192-4c66-ac31-d4ec53e5b29e-kube-api-access-dlnrm\") pod \"155fe462-1192-4c66-ac31-d4ec53e5b29e\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") "
Oct 29 23:31:31.028433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.028338    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-openshift-ca\") pod \"155fe462-1192-4c66-ac31-d4ec53e5b29e\" (UID: \"155fe462-1192-4c66-ac31-d4ec53e5b29e\") "
Oct 29 23:31:31.028810 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.028780    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-openshift-ca" pod "155fe462-1192-4c66-ac31-d4ec53e5b29e" (UID: "155fe462-1192-4c66-ac31-d4ec53e5b29e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:31.029656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.029118    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:31.030671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.030644    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-xks-ca" pod "155fe462-1192-4c66-ac31-d4ec53e5b29e" (UID: "155fe462-1192-4c66-ac31-d4ec53e5b29e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:31.031439 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.031419    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155fe462-1192-4c66-ac31-d4ec53e5b29e-kube-api-access-dlnrm" pod "155fe462-1192-4c66-ac31-d4ec53e5b29e" (UID: "155fe462-1192-4c66-ac31-d4ec53e5b29e"). InnerVolumeSpecName "kube-api-access-dlnrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:31.131257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.131219    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155fe462-1192-4c66-ac31-d4ec53e5b29e-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:31.131257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.131258    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlnrm\" (UniqueName: \"kubernetes.io/projected/155fe462-1192-4c66-ac31-d4ec53e5b29e-kube-api-access-dlnrm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:31.798739 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.798708    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"
Oct 29 23:31:31.798908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.798748    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn" event={"ID":"155fe462-1192-4c66-ac31-d4ec53e5b29e","Type":"ContainerDied","Data":"5abb94f3f48b9aa1e17cff25b2f8392cb7a2600a34728f99b8c5802c2c6bca7f"}
Oct 29 23:31:31.798908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:31.798803    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5abb94f3f48b9aa1e17cff25b2f8392cb7a2600a34728f99b8c5802c2c6bca7f"
Oct 29 23:31:32.414686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.414650    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"]
Oct 29 23:31:32.415286 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.415270    6203 cpu_manager.go:414] "containerMap: removing container" podUID="155fe462-1192-4c66-ac31-d4ec53e5b29e" containerName="revoke-keys"
Oct 29 23:31:32.415335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.415286    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155fe462-1192-4c66-ac31-d4ec53e5b29e" containerName="revoke-keys"
Oct 29 23:31:32.415426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.415416    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="155fe462-1192-4c66-ac31-d4ec53e5b29e" containerName="revoke-keys"
Oct 29 23:31:32.418922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.418904    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.439378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.439349    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"]
Oct 29 23:31:32.450252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.450225    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:32.450423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.450279    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:31:32.450506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.450424    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:31:32.450506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.450485    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:31:32.450812 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.450783    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.451060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.451035    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.451251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.451225    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqcf\" (UniqueName: \"kubernetes.io/projected/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-kube-api-access-zpqcf\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.552574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.552533    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.552747 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.552634    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqcf\" (UniqueName: \"kubernetes.io/projected/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-kube-api-access-zpqcf\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.552747 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.552679    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.553379 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.553348    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.554862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.554843    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.579136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.579110    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqcf\" (UniqueName: \"kubernetes.io/projected/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-kube-api-access-zpqcf\") pod \"maas-api-revoke-sub-ae3c9ebb-949tb\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:32.733804 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:32.733771    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:33.039261 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:33.039226    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e11d63_f475_428e_a5d9_6ad78d4b35ac.slice/crio-4cba66a018aa8630af7f75cd3a63e3f044f66585b723c500a5717823ba903d47 WatchSource:0}: Error finding container 4cba66a018aa8630af7f75cd3a63e3f044f66585b723c500a5717823ba903d47: Status 404 returned error can't find the container with id 4cba66a018aa8630af7f75cd3a63e3f044f66585b723c500a5717823ba903d47
Oct 29 23:31:33.039923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:33.039866    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"]
Oct 29 23:31:33.822046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:33.822008    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb" event={"ID":"e1e11d63-f475-428e-a5d9-6ad78d4b35ac","Type":"ContainerStarted","Data":"e2bf68a8a87670eac622ebce8366f86ca108cc0ebf9703486eff1df4d8b037d3"}
Oct 29 23:31:33.822451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:33.822067    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb" event={"ID":"e1e11d63-f475-428e-a5d9-6ad78d4b35ac","Type":"ContainerStarted","Data":"4cba66a018aa8630af7f75cd3a63e3f044f66585b723c500a5717823ba903d47"}
Oct 29 23:31:33.892700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:33.892634    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb" podStartSLOduration=1.892613777 podStartE2EDuration="1.892613777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:33.887932898 +0000 UTC m=+1071.992193285" watchObservedRunningTime="2026-10-29 23:31:33.892613777 +0000 UTC m=+1071.996874167"
Oct 29 23:31:34.832230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:34.832196    6203 generic.go:356] "Generic (PLEG): container finished" podID="e1e11d63-f475-428e-a5d9-6ad78d4b35ac" containerID="e2bf68a8a87670eac622ebce8366f86ca108cc0ebf9703486eff1df4d8b037d3" exitCode=0
Oct 29 23:31:34.832713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:34.832254    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb" event={"ID":"e1e11d63-f475-428e-a5d9-6ad78d4b35ac","Type":"ContainerDied","Data":"e2bf68a8a87670eac622ebce8366f86ca108cc0ebf9703486eff1df4d8b037d3"}
Oct 29 23:31:35.683594 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.683558    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"]
Oct 29 23:31:35.688071 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.688054    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.711659 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.711624    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"]
Oct 29 23:31:35.799822 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.799782    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.800039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.799967    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpcq6\" (UniqueName: \"kubernetes.io/projected/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-kube-api-access-mpcq6\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.800125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.800111    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.902393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.901514    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.902393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.901651    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpcq6\" (UniqueName: \"kubernetes.io/projected/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-kube-api-access-mpcq6\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.902393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.901775    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.902926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.902848    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.905528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.905484    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.934802 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.934724    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpcq6\" (UniqueName: \"kubernetes.io/projected/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-kube-api-access-mpcq6\") pod \"maas-api-revoke-sub-a20e476d-7fxcd\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:35.999035 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:35.999009    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:36.001776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.001756    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:36.103618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.103584    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-openshift-ca\") pod \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") "
Oct 29 23:31:36.103734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.103664    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-xks-ca\") pod \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") "
Oct 29 23:31:36.103734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.103706    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-kube-api-access-zpqcf\" (UniqueName: \"kubernetes.io/projected/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-kube-api-access-zpqcf\") pod \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\" (UID: \"e1e11d63-f475-428e-a5d9-6ad78d4b35ac\") "
Oct 29 23:31:36.106848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.105736    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-openshift-ca" pod "e1e11d63-f475-428e-a5d9-6ad78d4b35ac" (UID: "e1e11d63-f475-428e-a5d9-6ad78d4b35ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:36.108243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.108217    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-kube-api-access-zpqcf" pod "e1e11d63-f475-428e-a5d9-6ad78d4b35ac" (UID: "e1e11d63-f475-428e-a5d9-6ad78d4b35ac"). InnerVolumeSpecName "kube-api-access-zpqcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:36.108946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.108920    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-xks-ca" pod "e1e11d63-f475-428e-a5d9-6ad78d4b35ac" (UID: "e1e11d63-f475-428e-a5d9-6ad78d4b35ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:36.206315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.206239    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:36.206315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.206265    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqcf\" (UniqueName: \"kubernetes.io/projected/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-kube-api-access-zpqcf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:36.206315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.206275    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e11d63-f475-428e-a5d9-6ad78d4b35ac-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:36.330654 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:36.330621    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f622b3_5c02_40f0_aeca_7dcf14a8b0f6.slice/crio-a9e3f7680a586ed3dd6fa7bb1aea9f899af8a9588e03ea692f513d3f4e4d12ff WatchSource:0}: Error finding container a9e3f7680a586ed3dd6fa7bb1aea9f899af8a9588e03ea692f513d3f4e4d12ff: Status 404 returned error can't find the container with id a9e3f7680a586ed3dd6fa7bb1aea9f899af8a9588e03ea692f513d3f4e4d12ff
Oct 29 23:31:36.331372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.331349    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"]
Oct 29 23:31:36.847467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.847380    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"
Oct 29 23:31:36.847626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.847381    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb" event={"ID":"e1e11d63-f475-428e-a5d9-6ad78d4b35ac","Type":"ContainerDied","Data":"4cba66a018aa8630af7f75cd3a63e3f044f66585b723c500a5717823ba903d47"}
Oct 29 23:31:36.847626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.847521    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cba66a018aa8630af7f75cd3a63e3f044f66585b723c500a5717823ba903d47"
Oct 29 23:31:36.849240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.849213    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd" event={"ID":"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6","Type":"ContainerStarted","Data":"e9c06ef9ca789f8bc2e54cdd6cf1a4c6b032cd6a705a90e81563be867d25487a"}
Oct 29 23:31:36.849383 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.849262    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd" event={"ID":"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6","Type":"ContainerStarted","Data":"a9e3f7680a586ed3dd6fa7bb1aea9f899af8a9588e03ea692f513d3f4e4d12ff"}
Oct 29 23:31:36.898800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:36.898745    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd" podStartSLOduration=1.898727708 podStartE2EDuration="1.898727708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:36.895076686 +0000 UTC m=+1074.999337072" watchObservedRunningTime="2026-10-29 23:31:36.898727708 +0000 UTC m=+1075.002988094"
Oct 29 23:31:37.857987 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:37.857948    6203 generic.go:356] "Generic (PLEG): container finished" podID="31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" containerID="e9c06ef9ca789f8bc2e54cdd6cf1a4c6b032cd6a705a90e81563be867d25487a" exitCode=0
Oct 29 23:31:37.858398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:37.858027    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd" event={"ID":"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6","Type":"ContainerDied","Data":"e9c06ef9ca789f8bc2e54cdd6cf1a4c6b032cd6a705a90e81563be867d25487a"}
Oct 29 23:31:39.013944 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.013917    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:39.146004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.145926    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-xks-ca\") pod \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") "
Oct 29 23:31:39.146004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.145997    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-openshift-ca\") pod \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") "
Oct 29 23:31:39.146185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.146102    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-kube-api-access-mpcq6\" (UniqueName: \"kubernetes.io/projected/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-kube-api-access-mpcq6\") pod \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\" (UID: \"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6\") "
Oct 29 23:31:39.146426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.146400    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-openshift-ca" pod "31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" (UID: "31f622b3-5c02-40f0-aeca-7dcf14a8b0f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:39.146872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.146846    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:39.148181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.148160    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-xks-ca" pod "31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" (UID: "31f622b3-5c02-40f0-aeca-7dcf14a8b0f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:39.148314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.148293    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-kube-api-access-mpcq6" pod "31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" (UID: "31f622b3-5c02-40f0-aeca-7dcf14a8b0f6"). InnerVolumeSpecName "kube-api-access-mpcq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:39.248369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.248334    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:39.248369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.248365    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpcq6\" (UniqueName: \"kubernetes.io/projected/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6-kube-api-access-mpcq6\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:39.873297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.873261    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"
Oct 29 23:31:39.873297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.873284    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd" event={"ID":"31f622b3-5c02-40f0-aeca-7dcf14a8b0f6","Type":"ContainerDied","Data":"a9e3f7680a586ed3dd6fa7bb1aea9f899af8a9588e03ea692f513d3f4e4d12ff"}
Oct 29 23:31:39.873556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:39.873332    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9e3f7680a586ed3dd6fa7bb1aea9f899af8a9588e03ea692f513d3f4e4d12ff"
Oct 29 23:31:39.925149 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:31:39.925109    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:31:41.283775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.283735    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"]
Oct 29 23:31:41.284424 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.284407    6203 cpu_manager.go:414] "containerMap: removing container" podUID="31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" containerName="revoke-keys"
Oct 29 23:31:41.284469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.284425    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" containerName="revoke-keys"
Oct 29 23:31:41.284469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.284443    6203 cpu_manager.go:414] "containerMap: removing container" podUID="e1e11d63-f475-428e-a5d9-6ad78d4b35ac" containerName="revoke-keys"
Oct 29 23:31:41.284469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.284449    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e11d63-f475-428e-a5d9-6ad78d4b35ac" containerName="revoke-keys"
Oct 29 23:31:41.284556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.284530    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" containerName="revoke-keys"
Oct 29 23:31:41.284556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.284540    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e11d63-f475-428e-a5d9-6ad78d4b35ac" containerName="revoke-keys"
Oct 29 23:31:41.288315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.288298    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.308758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.308727    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"]
Oct 29 23:31:41.318592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.318560    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:41.318592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.318579    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:31:41.319958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.319944    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:31:41.320055 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.319978    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:31:41.375065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.375024    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.375065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.375064    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.375343 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.375317    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f742b\" (UniqueName: \"kubernetes.io/projected/efe9bc56-3e57-4bb7-8144-5dbca79e7379-kube-api-access-f742b\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.476740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.476698    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f742b\" (UniqueName: \"kubernetes.io/projected/efe9bc56-3e57-4bb7-8144-5dbca79e7379-kube-api-access-f742b\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.476978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.476776    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.476978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.476796    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.477495 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.477466    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.479230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.479203    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.504583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.504554    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f742b\" (UniqueName: \"kubernetes.io/projected/efe9bc56-3e57-4bb7-8144-5dbca79e7379-kube-api-access-f742b\") pod \"maas-api-revoke-sub-276d646a-tlm6s\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.602974 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.602850    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:41.910650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:41.910620    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"]
Oct 29 23:31:41.911061 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:41.911028    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe9bc56_3e57_4bb7_8144_5dbca79e7379.slice/crio-22f33d573a455f4dec5100dca09317d7f405b9d88d2447e7721c984276744c06 WatchSource:0}: Error finding container 22f33d573a455f4dec5100dca09317d7f405b9d88d2447e7721c984276744c06: Status 404 returned error can't find the container with id 22f33d573a455f4dec5100dca09317d7f405b9d88d2447e7721c984276744c06
Oct 29 23:31:42.897619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:42.897578    6203 generic.go:356] "Generic (PLEG): container finished" podID="efe9bc56-3e57-4bb7-8144-5dbca79e7379" containerID="2912293498a1f72df0cad8dc094e2ed7c80564dfdc68c0417ce7d50016d42d63" exitCode=0
Oct 29 23:31:42.898210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:42.897661    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s" event={"ID":"efe9bc56-3e57-4bb7-8144-5dbca79e7379","Type":"ContainerDied","Data":"2912293498a1f72df0cad8dc094e2ed7c80564dfdc68c0417ce7d50016d42d63"}
Oct 29 23:31:42.898210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:42.897731    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s" event={"ID":"efe9bc56-3e57-4bb7-8144-5dbca79e7379","Type":"ContainerStarted","Data":"22f33d573a455f4dec5100dca09317d7f405b9d88d2447e7721c984276744c06"}
Oct 29 23:31:44.064281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.064253    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:44.213073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.213040    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-xks-ca\") pod \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") "
Oct 29 23:31:44.213275 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.213217    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-openshift-ca\") pod \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") "
Oct 29 23:31:44.213275 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.213273    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe9bc56-3e57-4bb7-8144-5dbca79e7379-kube-api-access-f742b\" (UniqueName: \"kubernetes.io/projected/efe9bc56-3e57-4bb7-8144-5dbca79e7379-kube-api-access-f742b\") pod \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\" (UID: \"efe9bc56-3e57-4bb7-8144-5dbca79e7379\") "
Oct 29 23:31:44.213952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.213575    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-openshift-ca" pod "efe9bc56-3e57-4bb7-8144-5dbca79e7379" (UID: "efe9bc56-3e57-4bb7-8144-5dbca79e7379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:44.213952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.213725    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:44.215252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.215231    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-xks-ca" pod "efe9bc56-3e57-4bb7-8144-5dbca79e7379" (UID: "efe9bc56-3e57-4bb7-8144-5dbca79e7379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:44.215514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.215486    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe9bc56-3e57-4bb7-8144-5dbca79e7379-kube-api-access-f742b" pod "efe9bc56-3e57-4bb7-8144-5dbca79e7379" (UID: "efe9bc56-3e57-4bb7-8144-5dbca79e7379"). InnerVolumeSpecName "kube-api-access-f742b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:44.315400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.315366    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f742b\" (UniqueName: \"kubernetes.io/projected/efe9bc56-3e57-4bb7-8144-5dbca79e7379-kube-api-access-f742b\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:44.315400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.315397    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe9bc56-3e57-4bb7-8144-5dbca79e7379-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:44.912699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.912670    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"
Oct 29 23:31:44.912913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.912688    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s" event={"ID":"efe9bc56-3e57-4bb7-8144-5dbca79e7379","Type":"ContainerDied","Data":"22f33d573a455f4dec5100dca09317d7f405b9d88d2447e7721c984276744c06"}
Oct 29 23:31:44.912913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:44.912752    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f33d573a455f4dec5100dca09317d7f405b9d88d2447e7721c984276744c06"
Oct 29 23:31:47.069903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:47.066961    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:47.212938 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:47.208926    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:47.245049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:47.245007    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:47.676012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:47.675962    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:47.900397 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:47.900350    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.059791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.059686    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.060475 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.060444    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.085570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.082328    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.085570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.083139    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.116575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.108773    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:31:48.217018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.216972    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.217677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.217652    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:31:48.242187 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.242122    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:31:48.258906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.245949    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.258906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.246348    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:31:48.612522 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.611998    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:31:48.612522 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.612094    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.703903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.697686    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.703903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.699241    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:31:48.918129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.906992    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:31:48.918129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.907051    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.918129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.913962    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:48.918129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:48.914277    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:31:49.062265 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.060986    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:49.062265 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.061313    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:31:49.101900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.092126    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:49.101900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.092392    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:31:49.101900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.092622    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:31:49.103699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.103675    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:31:49.106040 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.106020    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:31:49.421962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.421255    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:49.728995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.710373    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:31:49.728995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.714958    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:31:49.767845 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.767798    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:31:49.829901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.825773    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:31:49.890701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.890670    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:31:49.917406 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:49.917380    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:31:50.650074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.650036    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"]
Oct 29 23:31:50.650994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.650970    6203 cpu_manager.go:414] "containerMap: removing container" podUID="efe9bc56-3e57-4bb7-8144-5dbca79e7379" containerName="revoke-keys"
Oct 29 23:31:50.650994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.650990    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe9bc56-3e57-4bb7-8144-5dbca79e7379" containerName="revoke-keys"
Oct 29 23:31:50.651186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.651095    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe9bc56-3e57-4bb7-8144-5dbca79e7379" containerName="revoke-keys"
Oct 29 23:31:50.655390 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.655364    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.677917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.677006    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:31:50.677917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.677075    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:31:50.682707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.682679    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"]
Oct 29 23:31:50.688435 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.688414    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:31:50.688541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.688502    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:31:50.737070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.737040    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"]
Oct 29 23:31:50.741417 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.741394    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.760699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.760669    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.760846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.760825    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.761060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.761034    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/c499ac31-e352-431e-88e0-32df1bd0d90e-kube-api-access-h7s6g\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.765074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.765053    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"]
Oct 29 23:31:50.862643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.862602    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/72d1ab42-5bf8-48a5-8988-e2f8729911ab-kube-api-access-jwcv8\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.862814 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.862667    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/c499ac31-e352-431e-88e0-32df1bd0d90e-kube-api-access-h7s6g\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.862814 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.862735    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.862982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.862848    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.863045 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.862982    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.863101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.863070    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.863793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.863767    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.865608 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.865582    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.889030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.889006    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/c499ac31-e352-431e-88e0-32df1bd0d90e-kube-api-access-h7s6g\") pod \"maas-api-revoke-sub-e14aef47-sm2rd\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.964432 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.964401    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/72d1ab42-5bf8-48a5-8988-e2f8729911ab-kube-api-access-jwcv8\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.964590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.964443    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.964590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.964499    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.965277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.965249    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.966793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.966772    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:50.974872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.974851    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:50.990485 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:50.990466    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/72d1ab42-5bf8-48a5-8988-e2f8729911ab-kube-api-access-jwcv8\") pod \"maas-api-revoke-sub-0dbae4be-r5ntw\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:51.055391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:51.055360    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:51.301376 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:51.301344    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc499ac31_e352_431e_88e0_32df1bd0d90e.slice/crio-dfef557fdbe65b66063ee39c74e0ba5e72b0001ddef5cefdd192aac8dcec34bd WatchSource:0}: Error finding container dfef557fdbe65b66063ee39c74e0ba5e72b0001ddef5cefdd192aac8dcec34bd: Status 404 returned error can't find the container with id dfef557fdbe65b66063ee39c74e0ba5e72b0001ddef5cefdd192aac8dcec34bd
Oct 29 23:31:51.302774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:51.302753    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"]
Oct 29 23:31:51.395649 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:51.395618    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d1ab42_5bf8_48a5_8988_e2f8729911ab.slice/crio-81e4d2071cf24f9b4369c03414a2a5a4bf2a9605a08bd15ad1ead0cf07154d0a WatchSource:0}: Error finding container 81e4d2071cf24f9b4369c03414a2a5a4bf2a9605a08bd15ad1ead0cf07154d0a: Status 404 returned error can't find the container with id 81e4d2071cf24f9b4369c03414a2a5a4bf2a9605a08bd15ad1ead0cf07154d0a
Oct 29 23:31:51.396240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:51.396215    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"]
Oct 29 23:31:52.197410 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:52.197377    6203 generic.go:356] "Generic (PLEG): container finished" podID="c499ac31-e352-431e-88e0-32df1bd0d90e" containerID="aa3a203d58a9e876c613a5cee01bbbd6290fe50e892a9f27c229e28781e1d206" exitCode=0
Oct 29 23:31:52.197863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:52.197461    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd" event={"ID":"c499ac31-e352-431e-88e0-32df1bd0d90e","Type":"ContainerDied","Data":"aa3a203d58a9e876c613a5cee01bbbd6290fe50e892a9f27c229e28781e1d206"}
Oct 29 23:31:52.197863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:52.197543    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd" event={"ID":"c499ac31-e352-431e-88e0-32df1bd0d90e","Type":"ContainerStarted","Data":"dfef557fdbe65b66063ee39c74e0ba5e72b0001ddef5cefdd192aac8dcec34bd"}
Oct 29 23:31:52.199222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:52.199197    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw" event={"ID":"72d1ab42-5bf8-48a5-8988-e2f8729911ab","Type":"ContainerStarted","Data":"c4ff91fe3a8590b6311b9e5001e14ccb6619f022d8862f9d48e55000045bdf4e"}
Oct 29 23:31:52.199365 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:52.199237    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw" event={"ID":"72d1ab42-5bf8-48a5-8988-e2f8729911ab","Type":"ContainerStarted","Data":"81e4d2071cf24f9b4369c03414a2a5a4bf2a9605a08bd15ad1ead0cf07154d0a"}
Oct 29 23:31:52.290132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:52.290079    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw" podStartSLOduration=2.290067049 podStartE2EDuration="2.290067049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:52.28810078 +0000 UTC m=+1090.392361163" watchObservedRunningTime="2026-10-29 23:31:52.290067049 +0000 UTC m=+1090.394327475"
Oct 29 23:31:53.207536 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.207502    6203 generic.go:356] "Generic (PLEG): container finished" podID="72d1ab42-5bf8-48a5-8988-e2f8729911ab" containerID="c4ff91fe3a8590b6311b9e5001e14ccb6619f022d8862f9d48e55000045bdf4e" exitCode=0
Oct 29 23:31:53.208017 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.207594    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw" event={"ID":"72d1ab42-5bf8-48a5-8988-e2f8729911ab","Type":"ContainerDied","Data":"c4ff91fe3a8590b6311b9e5001e14ccb6619f022d8862f9d48e55000045bdf4e"}
Oct 29 23:31:53.368469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.368441    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:53.495454 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.495366    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-xks-ca\") pod \"c499ac31-e352-431e-88e0-32df1bd0d90e\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") "
Oct 29 23:31:53.495454 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.495415    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-openshift-ca\") pod \"c499ac31-e352-431e-88e0-32df1bd0d90e\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") "
Oct 29 23:31:53.495689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.495497    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c499ac31-e352-431e-88e0-32df1bd0d90e-kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/c499ac31-e352-431e-88e0-32df1bd0d90e-kube-api-access-h7s6g\") pod \"c499ac31-e352-431e-88e0-32df1bd0d90e\" (UID: \"c499ac31-e352-431e-88e0-32df1bd0d90e\") "
Oct 29 23:31:53.495837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.495805    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-openshift-ca" pod "c499ac31-e352-431e-88e0-32df1bd0d90e" (UID: "c499ac31-e352-431e-88e0-32df1bd0d90e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:53.496173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.496156    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:53.497597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.497573    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-xks-ca" pod "c499ac31-e352-431e-88e0-32df1bd0d90e" (UID: "c499ac31-e352-431e-88e0-32df1bd0d90e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:53.497678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.497653    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c499ac31-e352-431e-88e0-32df1bd0d90e-kube-api-access-h7s6g" pod "c499ac31-e352-431e-88e0-32df1bd0d90e" (UID: "c499ac31-e352-431e-88e0-32df1bd0d90e"). InnerVolumeSpecName "kube-api-access-h7s6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:53.597755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.597714    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c499ac31-e352-431e-88e0-32df1bd0d90e-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:53.597755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.597750    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7s6g\" (UniqueName: \"kubernetes.io/projected/c499ac31-e352-431e-88e0-32df1bd0d90e-kube-api-access-h7s6g\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:53.805960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.805861    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"]
Oct 29 23:31:53.806695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.806670    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c499ac31-e352-431e-88e0-32df1bd0d90e" containerName="revoke-keys"
Oct 29 23:31:53.806695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.806696    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c499ac31-e352-431e-88e0-32df1bd0d90e" containerName="revoke-keys"
Oct 29 23:31:53.806850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.806844    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c499ac31-e352-431e-88e0-32df1bd0d90e" containerName="revoke-keys"
Oct 29 23:31:53.811189 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.811161    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:53.835581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.835554    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"]
Oct 29 23:31:53.974577 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.974505    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"]
Oct 29 23:31:53.978849 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:53.978833    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.004034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004002    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004049    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004139    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004186    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004206    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004234    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004275    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004296    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.004472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.004334    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2nj\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-kube-api-access-dz2nj\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.007054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.007020    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"]
Oct 29 23:31:54.105525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.105439    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2nj\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-kube-api-access-dz2nj\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.105671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.105615    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.105737 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.105720    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25c6\" (UniqueName: \"kubernetes.io/projected/ba1fa1fa-c139-4b83-bb80-f18e1c104860-kube-api-access-r25c6\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.105777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.105760    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.105838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.105814    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.105910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.105859    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.105967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.105939    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106102 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106073    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106129    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106183    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106254    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106295    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106440    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-certs\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106704    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-credential-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106738    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-data\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.106960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.106844    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istiod-ca-cert\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.107065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.107045    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-socket\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.108388 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.108362    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-envoy\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.108503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.108389    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-podinfo\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.131297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.131248    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2nj\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-kube-api-access-dz2nj\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.138331 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.138310    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-token\") pod \"e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") " pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.207358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.207325    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25c6\" (UniqueName: \"kubernetes.io/projected/ba1fa1fa-c139-4b83-bb80-f18e1c104860-kube-api-access-r25c6\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.207534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.207367    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.207534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.207398    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.208116 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.208082    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.209740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.209720    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.215966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.215947    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"
Oct 29 23:31:54.216057 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.215983    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd" event={"ID":"c499ac31-e352-431e-88e0-32df1bd0d90e","Type":"ContainerDied","Data":"dfef557fdbe65b66063ee39c74e0ba5e72b0001ddef5cefdd192aac8dcec34bd"}
Oct 29 23:31:54.216057 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.216030    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfef557fdbe65b66063ee39c74e0ba5e72b0001ddef5cefdd192aac8dcec34bd"
Oct 29 23:31:54.243733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.243693    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25c6\" (UniqueName: \"kubernetes.io/projected/ba1fa1fa-c139-4b83-bb80-f18e1c104860-kube-api-access-r25c6\") pod \"maas-api-revoke-sub-fb669009-d4rh7\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.302733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.297918    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:54.420922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.420894    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:54.430262 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.430240    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:54.524861 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.524813    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d1ab42-5bf8-48a5-8988-e2f8729911ab-kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/72d1ab42-5bf8-48a5-8988-e2f8729911ab-kube-api-access-jwcv8\") pod \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") "
Oct 29 23:31:54.525070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.525050    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-openshift-ca\") pod \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") "
Oct 29 23:31:54.525172 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.525152    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-xks-ca\") pod \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\" (UID: \"72d1ab42-5bf8-48a5-8988-e2f8729911ab\") "
Oct 29 23:31:54.525451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.525423    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-openshift-ca" pod "72d1ab42-5bf8-48a5-8988-e2f8729911ab" (UID: "72d1ab42-5bf8-48a5-8988-e2f8729911ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:54.525803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.525780    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:54.527788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.527759    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d1ab42-5bf8-48a5-8988-e2f8729911ab-kube-api-access-jwcv8" pod "72d1ab42-5bf8-48a5-8988-e2f8729911ab" (UID: "72d1ab42-5bf8-48a5-8988-e2f8729911ab"). InnerVolumeSpecName "kube-api-access-jwcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:54.528832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.528678    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-xks-ca" pod "72d1ab42-5bf8-48a5-8988-e2f8729911ab" (UID: "72d1ab42-5bf8-48a5-8988-e2f8729911ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:54.626886 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.626826    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d1ab42-5bf8-48a5-8988-e2f8729911ab-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:54.626886 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.626870    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwcv8\" (UniqueName: \"kubernetes.io/projected/72d1ab42-5bf8-48a5-8988-e2f8729911ab-kube-api-access-jwcv8\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:54.635451 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:54.635411    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1fa1fa_c139_4b83_bb80_f18e1c104860.slice/crio-5fd875ff837f7e74ff463c676e132fffce636f62b0e4a2d7b01099d906b2e59c WatchSource:0}: Error finding container 5fd875ff837f7e74ff463c676e132fffce636f62b0e4a2d7b01099d906b2e59c: Status 404 returned error can't find the container with id 5fd875ff837f7e74ff463c676e132fffce636f62b0e4a2d7b01099d906b2e59c
Oct 29 23:31:54.636391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.636362    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"]
Oct 29 23:31:54.756252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.756224    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"]
Oct 29 23:31:54.756369 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:54.756249    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c99ab68_24ca_4d67_9c01_ec5bfc1146e7.slice/crio-76dd70be76c5a754b80f3d28d7ff43177e9dbe8545883105a2718755b4bf9deb WatchSource:0}: Error finding container 76dd70be76c5a754b80f3d28d7ff43177e9dbe8545883105a2718755b4bf9deb: Status 404 returned error can't find the container with id 76dd70be76c5a754b80f3d28d7ff43177e9dbe8545883105a2718755b4bf9deb
Oct 29 23:31:54.759134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.759089    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:54.759244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.759185    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:54.759244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:54.759230    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:31:55.102192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.102157    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"]
Oct 29 23:31:55.102848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.102833    6203 cpu_manager.go:414] "containerMap: removing container" podUID="72d1ab42-5bf8-48a5-8988-e2f8729911ab" containerName="revoke-keys"
Oct 29 23:31:55.102920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.102850    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d1ab42-5bf8-48a5-8988-e2f8729911ab" containerName="revoke-keys"
Oct 29 23:31:55.102965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.102951    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d1ab42-5bf8-48a5-8988-e2f8729911ab" containerName="revoke-keys"
Oct 29 23:31:55.106948 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.106923    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.133788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.133759    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.133971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.133948    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.134039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.134001    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bwn\" (UniqueName: \"kubernetes.io/projected/b580b2e2-83b9-44b0-b807-c71126fd8864-kube-api-access-s8bwn\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.143912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.143869    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"]
Oct 29 23:31:55.226176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.226082    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" event={"ID":"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7","Type":"ContainerStarted","Data":"24b9191d859b432356f985f2404986cdd79938be01c274b4bdeec51cbc39aa3e"}
Oct 29 23:31:55.226706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.226668    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" event={"ID":"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7","Type":"ContainerStarted","Data":"76dd70be76c5a754b80f3d28d7ff43177e9dbe8545883105a2718755b4bf9deb"}
Oct 29 23:31:55.227848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.227823    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7" event={"ID":"ba1fa1fa-c139-4b83-bb80-f18e1c104860","Type":"ContainerStarted","Data":"7518338df29d17390e10bcf05fd2be5d8ae90994c8691cd60a3f33ff3e5eff20"}
Oct 29 23:31:55.227980 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.227872    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7" event={"ID":"ba1fa1fa-c139-4b83-bb80-f18e1c104860","Type":"ContainerStarted","Data":"5fd875ff837f7e74ff463c676e132fffce636f62b0e4a2d7b01099d906b2e59c"}
Oct 29 23:31:55.230447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.230422    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw" event={"ID":"72d1ab42-5bf8-48a5-8988-e2f8729911ab","Type":"ContainerDied","Data":"81e4d2071cf24f9b4369c03414a2a5a4bf2a9605a08bd15ad1ead0cf07154d0a"}
Oct 29 23:31:55.230447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.230446    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"
Oct 29 23:31:55.230597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.230465    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e4d2071cf24f9b4369c03414a2a5a4bf2a9605a08bd15ad1ead0cf07154d0a"
Oct 29 23:31:55.235268 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.235238    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.235440 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.235417    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.235536 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.235485    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bwn\" (UniqueName: \"kubernetes.io/projected/b580b2e2-83b9-44b0-b807-c71126fd8864-kube-api-access-s8bwn\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.236469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.236438    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.237995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.237972    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.263397 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.263364    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bwn\" (UniqueName: \"kubernetes.io/projected/b580b2e2-83b9-44b0-b807-c71126fd8864-kube-api-access-s8bwn\") pod \"maas-api-revoke-sub-5026caab-8ngvg\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.283736 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.283674    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podStartSLOduration=2.283656862 podStartE2EDuration="2.283656862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:55.279376499 +0000 UTC m=+1093.383636885" watchObservedRunningTime="2026-10-29 23:31:55.283656862 +0000 UTC m=+1093.387917247"
Oct 29 23:31:55.325188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.325129    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7" podStartSLOduration=2.325115156 podStartE2EDuration="2.325115156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:55.32442706 +0000 UTC m=+1093.428687454" watchObservedRunningTime="2026-10-29 23:31:55.325115156 +0000 UTC m=+1093.429375541"
Oct 29 23:31:55.421734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.421701    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:55.436069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.434186    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:31:55.436069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.435707    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 23:31:55.436069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.435758    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 23:31:55.757734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:55.757704    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"]
Oct 29 23:31:55.757734 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:31:55.757717    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb580b2e2_83b9_44b0_b807_c71126fd8864.slice/crio-85249da18c91cb900f942e85a21601a629026c9a099b7177f78e06fbee80d7d4 WatchSource:0}: Error finding container 85249da18c91cb900f942e85a21601a629026c9a099b7177f78e06fbee80d7d4: Status 404 returned error can't find the container with id 85249da18c91cb900f942e85a21601a629026c9a099b7177f78e06fbee80d7d4
Oct 29 23:31:56.260687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:56.260651    6203 generic.go:356] "Generic (PLEG): container finished" podID="ba1fa1fa-c139-4b83-bb80-f18e1c104860" containerID="7518338df29d17390e10bcf05fd2be5d8ae90994c8691cd60a3f33ff3e5eff20" exitCode=0
Oct 29 23:31:56.261136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:56.260731    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7" event={"ID":"ba1fa1fa-c139-4b83-bb80-f18e1c104860","Type":"ContainerDied","Data":"7518338df29d17390e10bcf05fd2be5d8ae90994c8691cd60a3f33ff3e5eff20"}
Oct 29 23:31:56.262683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:56.262646    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg" event={"ID":"b580b2e2-83b9-44b0-b807-c71126fd8864","Type":"ContainerStarted","Data":"35dcd61b52525edc577a24f76fc58777e1e82a56f7bf6c60694106ab2b7ea363"}
Oct 29 23:31:56.262837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:56.262695    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg" event={"ID":"b580b2e2-83b9-44b0-b807-c71126fd8864","Type":"ContainerStarted","Data":"85249da18c91cb900f942e85a21601a629026c9a099b7177f78e06fbee80d7d4"}
Oct 29 23:31:56.361400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:56.361334    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg" podStartSLOduration=1.3613137069999999 podStartE2EDuration="1.361313707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:31:56.360338555 +0000 UTC m=+1094.464598944" watchObservedRunningTime="2026-10-29 23:31:56.361313707 +0000 UTC m=+1094.465574093"
Oct 29 23:31:56.460013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:56.459971    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 23:31:56.460229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:56.460029    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 23:31:57.271845 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.271809    6203 generic.go:356] "Generic (PLEG): container finished" podID="b580b2e2-83b9-44b0-b807-c71126fd8864" containerID="35dcd61b52525edc577a24f76fc58777e1e82a56f7bf6c60694106ab2b7ea363" exitCode=0
Oct 29 23:31:57.272333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.271919    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg" event={"ID":"b580b2e2-83b9-44b0-b807-c71126fd8864","Type":"ContainerDied","Data":"35dcd61b52525edc577a24f76fc58777e1e82a56f7bf6c60694106ab2b7ea363"}
Oct 29 23:31:57.459357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.459313    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 23:31:57.459531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.459386    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 23:31:57.498557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.498530    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:57.600574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.600481    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-xks-ca\") pod \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") "
Oct 29 23:31:57.600741 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.600598    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1fa1fa-c139-4b83-bb80-f18e1c104860-kube-api-access-r25c6\" (UniqueName: \"kubernetes.io/projected/ba1fa1fa-c139-4b83-bb80-f18e1c104860-kube-api-access-r25c6\") pod \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") "
Oct 29 23:31:57.600820 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.600808    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-openshift-ca\") pod \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\" (UID: \"ba1fa1fa-c139-4b83-bb80-f18e1c104860\") "
Oct 29 23:31:57.601488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.601456    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-openshift-ca" pod "ba1fa1fa-c139-4b83-bb80-f18e1c104860" (UID: "ba1fa1fa-c139-4b83-bb80-f18e1c104860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:57.603321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.603288    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-xks-ca" pod "ba1fa1fa-c139-4b83-bb80-f18e1c104860" (UID: "ba1fa1fa-c139-4b83-bb80-f18e1c104860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:57.603440 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.603345    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1fa1fa-c139-4b83-bb80-f18e1c104860-kube-api-access-r25c6" pod "ba1fa1fa-c139-4b83-bb80-f18e1c104860" (UID: "ba1fa1fa-c139-4b83-bb80-f18e1c104860"). InnerVolumeSpecName "kube-api-access-r25c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:57.702671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.702638    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:57.702671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.702670    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1fa1fa-c139-4b83-bb80-f18e1c104860-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:57.702671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:57.702679    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25c6\" (UniqueName: \"kubernetes.io/projected/ba1fa1fa-c139-4b83-bb80-f18e1c104860-kube-api-access-r25c6\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:58.280181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.280082    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"
Oct 29 23:31:58.280181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.280119    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7" event={"ID":"ba1fa1fa-c139-4b83-bb80-f18e1c104860","Type":"ContainerDied","Data":"5fd875ff837f7e74ff463c676e132fffce636f62b0e4a2d7b01099d906b2e59c"}
Oct 29 23:31:58.280181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.280176    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fd875ff837f7e74ff463c676e132fffce636f62b0e4a2d7b01099d906b2e59c"
Oct 29 23:31:58.456325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.456284    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 23:31:58.456488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.456373    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 23:31:58.461651 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.461628    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:58.524843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.524791    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-xks-ca\") pod \"b580b2e2-83b9-44b0-b807-c71126fd8864\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") "
Oct 29 23:31:58.525050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.524975    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-openshift-ca\") pod \"b580b2e2-83b9-44b0-b807-c71126fd8864\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") "
Oct 29 23:31:58.525050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.525019    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b580b2e2-83b9-44b0-b807-c71126fd8864-kube-api-access-s8bwn\" (UniqueName: \"kubernetes.io/projected/b580b2e2-83b9-44b0-b807-c71126fd8864-kube-api-access-s8bwn\") pod \"b580b2e2-83b9-44b0-b807-c71126fd8864\" (UID: \"b580b2e2-83b9-44b0-b807-c71126fd8864\") "
Oct 29 23:31:58.526406 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.526380    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-openshift-ca" pod "b580b2e2-83b9-44b0-b807-c71126fd8864" (UID: "b580b2e2-83b9-44b0-b807-c71126fd8864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:31:58.549095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.549017    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-xks-ca" pod "b580b2e2-83b9-44b0-b807-c71126fd8864" (UID: "b580b2e2-83b9-44b0-b807-c71126fd8864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:31:58.549408 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.549350    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b580b2e2-83b9-44b0-b807-c71126fd8864-kube-api-access-s8bwn" pod "b580b2e2-83b9-44b0-b807-c71126fd8864" (UID: "b580b2e2-83b9-44b0-b807-c71126fd8864"). InnerVolumeSpecName "kube-api-access-s8bwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:31:58.627181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.627148    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:58.627181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.627177    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b580b2e2-83b9-44b0-b807-c71126fd8864-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:58.627181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:58.627188    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8bwn\" (UniqueName: \"kubernetes.io/projected/b580b2e2-83b9-44b0-b807-c71126fd8864-kube-api-access-s8bwn\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:31:59.288220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:59.288192    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"
Oct 29 23:31:59.288607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:59.288219    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg" event={"ID":"b580b2e2-83b9-44b0-b807-c71126fd8864","Type":"ContainerDied","Data":"85249da18c91cb900f942e85a21601a629026c9a099b7177f78e06fbee80d7d4"}
Oct 29 23:31:59.288607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:59.288261    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85249da18c91cb900f942e85a21601a629026c9a099b7177f78e06fbee80d7d4"
Oct 29 23:31:59.449903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:59.448973    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 23:31:59.449903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:31:59.449049    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 23:32:00.453363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:00.453298    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 23:32:00.453856 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:00.453380    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 23:32:01.158492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.158458    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"]
Oct 29 23:32:01.159089 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.159075    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ba1fa1fa-c139-4b83-bb80-f18e1c104860" containerName="revoke-keys"
Oct 29 23:32:01.159147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.159092    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1fa1fa-c139-4b83-bb80-f18e1c104860" containerName="revoke-keys"
Oct 29 23:32:01.159147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.159128    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b580b2e2-83b9-44b0-b807-c71126fd8864" containerName="revoke-keys"
Oct 29 23:32:01.159147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.159133    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b580b2e2-83b9-44b0-b807-c71126fd8864" containerName="revoke-keys"
Oct 29 23:32:01.159258 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.159248    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b580b2e2-83b9-44b0-b807-c71126fd8864" containerName="revoke-keys"
Oct 29 23:32:01.159293 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.159265    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1fa1fa-c139-4b83-bb80-f18e1c104860" containerName="revoke-keys"
Oct 29 23:32:01.163443 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.163414    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.185939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.185896    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"]
Oct 29 23:32:01.193293 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.193027    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:32:01.193293 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.193138    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:32:01.196822 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.196798    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:01.197001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.196836    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:32:01.209735 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.209701    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"]
Oct 29 23:32:01.214604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.214581    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.240377 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.240349    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"]
Oct 29 23:32:01.283739 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.283707    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.283979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.283957    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5df\" (UniqueName: \"kubernetes.io/projected/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-kube-api-access-cq5df\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.284039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.284023    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.385629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.385582    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5df\" (UniqueName: \"kubernetes.io/projected/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-kube-api-access-cq5df\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.385855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.385650    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzgj\" (UniqueName: \"kubernetes.io/projected/8a19dd1e-502f-4d23-8666-a05938cef188-kube-api-access-ffzgj\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.385855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.385717    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.385996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.385922    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.386059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.386020    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.386195 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.386166    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.386613 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.386593    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.388300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.388275    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.412531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.412449    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5df\" (UniqueName: \"kubernetes.io/projected/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-kube-api-access-cq5df\") pod \"maas-api-revoke-sub-d73b451a-qbrnf\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.431475 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.431446    6203 patch_prober.go:29] interesting pod/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 23:32:01.431590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.431508    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 23:32:01.479953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.479911    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:01.487681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.487648    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.487817 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.487746    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzgj\" (UniqueName: \"kubernetes.io/projected/8a19dd1e-502f-4d23-8666-a05938cef188-kube-api-access-ffzgj\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.487978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.487956    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.488385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.488358    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.490337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.490313    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.514406 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.514370    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzgj\" (UniqueName: \"kubernetes.io/projected/8a19dd1e-502f-4d23-8666-a05938cef188-kube-api-access-ffzgj\") pod \"maas-api-revoke-sub-1fb483b5-5w49m\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.543598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.543488    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:01.811378 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:01.811345    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39c2cc6_0db7_4b32_9786_f5b911fa6e52.slice/crio-9a08b6920f5ad9cf7f6798cd3101b6e419f8a27fbfec9aa43bbd40b41fa72377 WatchSource:0}: Error finding container 9a08b6920f5ad9cf7f6798cd3101b6e419f8a27fbfec9aa43bbd40b41fa72377: Status 404 returned error can't find the container with id 9a08b6920f5ad9cf7f6798cd3101b6e419f8a27fbfec9aa43bbd40b41fa72377
Oct 29 23:32:01.812546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.812511    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"]
Oct 29 23:32:01.914655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:01.914623    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"]
Oct 29 23:32:01.918328 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:01.918295    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a19dd1e_502f_4d23_8666_a05938cef188.slice/crio-e43c3bffca268fb2bb05146b41cc9b54f001345f54b5c538db772afadc9b474d WatchSource:0}: Error finding container e43c3bffca268fb2bb05146b41cc9b54f001345f54b5c538db772afadc9b474d: Status 404 returned error can't find the container with id e43c3bffca268fb2bb05146b41cc9b54f001345f54b5c538db772afadc9b474d
Oct 29 23:32:02.315279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.315240    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m" event={"ID":"8a19dd1e-502f-4d23-8666-a05938cef188","Type":"ContainerStarted","Data":"b587ed9a4acd9f47485ab40e2511f07a742d4046fa21658a9c6b6e75a7fbe47c"}
Oct 29 23:32:02.315767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.315305    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m" event={"ID":"8a19dd1e-502f-4d23-8666-a05938cef188","Type":"ContainerStarted","Data":"e43c3bffca268fb2bb05146b41cc9b54f001345f54b5c538db772afadc9b474d"}
Oct 29 23:32:02.317526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.317486    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf" event={"ID":"b39c2cc6-0db7-4b32-9786-f5b911fa6e52","Type":"ContainerStarted","Data":"acf81e0eed2d9be4abab88d707c9ffc558c1e2b2870e1991343a50b740005efb"}
Oct 29 23:32:02.317626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.317550    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf" event={"ID":"b39c2cc6-0db7-4b32-9786-f5b911fa6e52","Type":"ContainerStarted","Data":"9a08b6920f5ad9cf7f6798cd3101b6e419f8a27fbfec9aa43bbd40b41fa72377"}
Oct 29 23:32:02.365170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.365117    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m" podStartSLOduration=1.365101721 podStartE2EDuration="1.365101721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:02.361129055 +0000 UTC m=+1100.465389441" watchObservedRunningTime="2026-10-29 23:32:02.365101721 +0000 UTC m=+1100.469362140"
Oct 29 23:32:02.411978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.411919    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf" podStartSLOduration=1.411905477 podStartE2EDuration="1.411905477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:02.408570169 +0000 UTC m=+1100.512830555" watchObservedRunningTime="2026-10-29 23:32:02.411905477 +0000 UTC m=+1100.516165857"
Oct 29 23:32:02.441748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.441714    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:32:02.443812 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.443783    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:32:02.444547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:02.444496    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:32:03.327173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:03.327141    6203 generic.go:356] "Generic (PLEG): container finished" podID="b39c2cc6-0db7-4b32-9786-f5b911fa6e52" containerID="acf81e0eed2d9be4abab88d707c9ffc558c1e2b2870e1991343a50b740005efb" exitCode=0
Oct 29 23:32:03.327616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:03.327225    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf" event={"ID":"b39c2cc6-0db7-4b32-9786-f5b911fa6e52","Type":"ContainerDied","Data":"acf81e0eed2d9be4abab88d707c9ffc558c1e2b2870e1991343a50b740005efb"}
Oct 29 23:32:03.329074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:03.329046    6203 generic.go:356] "Generic (PLEG): container finished" podID="8a19dd1e-502f-4d23-8666-a05938cef188" containerID="b587ed9a4acd9f47485ab40e2511f07a742d4046fa21658a9c6b6e75a7fbe47c" exitCode=0
Oct 29 23:32:03.329193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:03.329116    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m" event={"ID":"8a19dd1e-502f-4d23-8666-a05938cef188","Type":"ContainerDied","Data":"b587ed9a4acd9f47485ab40e2511f07a742d4046fa21658a9c6b6e75a7fbe47c"}
Oct 29 23:32:04.535343 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.535312    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:04.538629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.538607    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:04.639947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.639852    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-xks-ca\") pod \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") "
Oct 29 23:32:04.640129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.640002    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-kube-api-access-cq5df\" (UniqueName: \"kubernetes.io/projected/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-kube-api-access-cq5df\") pod \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") "
Oct 29 23:32:04.640657 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.640626    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-openshift-ca" pod "b39c2cc6-0db7-4b32-9786-f5b911fa6e52" (UID: "b39c2cc6-0db7-4b32-9786-f5b911fa6e52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:04.641142 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.641109    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-openshift-ca\") pod \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\" (UID: \"b39c2cc6-0db7-4b32-9786-f5b911fa6e52\") "
Oct 29 23:32:04.643591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.643563    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:04.664217 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.664172    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-kube-api-access-cq5df" pod "b39c2cc6-0db7-4b32-9786-f5b911fa6e52" (UID: "b39c2cc6-0db7-4b32-9786-f5b911fa6e52"). InnerVolumeSpecName "kube-api-access-cq5df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:04.669387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.669350    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-xks-ca" pod "b39c2cc6-0db7-4b32-9786-f5b911fa6e52" (UID: "b39c2cc6-0db7-4b32-9786-f5b911fa6e52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:04.744038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.743995    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a19dd1e-502f-4d23-8666-a05938cef188-kube-api-access-ffzgj\" (UniqueName: \"kubernetes.io/projected/8a19dd1e-502f-4d23-8666-a05938cef188-kube-api-access-ffzgj\") pod \"8a19dd1e-502f-4d23-8666-a05938cef188\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") "
Oct 29 23:32:04.744216 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.744078    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-xks-ca\") pod \"8a19dd1e-502f-4d23-8666-a05938cef188\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") "
Oct 29 23:32:04.744282 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.744219    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-openshift-ca\") pod \"8a19dd1e-502f-4d23-8666-a05938cef188\" (UID: \"8a19dd1e-502f-4d23-8666-a05938cef188\") "
Oct 29 23:32:04.744597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.744564    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-openshift-ca" pod "8a19dd1e-502f-4d23-8666-a05938cef188" (UID: "8a19dd1e-502f-4d23-8666-a05938cef188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:04.744750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.744633    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:04.744750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.744649    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5df\" (UniqueName: \"kubernetes.io/projected/b39c2cc6-0db7-4b32-9786-f5b911fa6e52-kube-api-access-cq5df\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:04.746239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.746214    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-xks-ca" pod "8a19dd1e-502f-4d23-8666-a05938cef188" (UID: "8a19dd1e-502f-4d23-8666-a05938cef188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:04.746338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.746261    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a19dd1e-502f-4d23-8666-a05938cef188-kube-api-access-ffzgj" pod "8a19dd1e-502f-4d23-8666-a05938cef188" (UID: "8a19dd1e-502f-4d23-8666-a05938cef188"). InnerVolumeSpecName "kube-api-access-ffzgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:04.845525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.845487    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:04.845525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.845520    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzgj\" (UniqueName: \"kubernetes.io/projected/8a19dd1e-502f-4d23-8666-a05938cef188-kube-api-access-ffzgj\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:04.845525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:04.845530    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a19dd1e-502f-4d23-8666-a05938cef188-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:05.346657 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.346625    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"
Oct 29 23:32:05.346657 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.346645    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m" event={"ID":"8a19dd1e-502f-4d23-8666-a05938cef188","Type":"ContainerDied","Data":"e43c3bffca268fb2bb05146b41cc9b54f001345f54b5c538db772afadc9b474d"}
Oct 29 23:32:05.346888 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.346702    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e43c3bffca268fb2bb05146b41cc9b54f001345f54b5c538db772afadc9b474d"
Oct 29 23:32:05.348286 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.348267    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"
Oct 29 23:32:05.348398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.348308    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf" event={"ID":"b39c2cc6-0db7-4b32-9786-f5b911fa6e52","Type":"ContainerDied","Data":"9a08b6920f5ad9cf7f6798cd3101b6e419f8a27fbfec9aa43bbd40b41fa72377"}
Oct 29 23:32:05.348398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.348356    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a08b6920f5ad9cf7f6798cd3101b6e419f8a27fbfec9aa43bbd40b41fa72377"
Oct 29 23:32:05.874778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.874743    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"]
Oct 29 23:32:05.875397 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.875382    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b39c2cc6-0db7-4b32-9786-f5b911fa6e52" containerName="revoke-keys"
Oct 29 23:32:05.875397 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.875397    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39c2cc6-0db7-4b32-9786-f5b911fa6e52" containerName="revoke-keys"
Oct 29 23:32:05.875523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.875417    6203 cpu_manager.go:414] "containerMap: removing container" podUID="8a19dd1e-502f-4d23-8666-a05938cef188" containerName="revoke-keys"
Oct 29 23:32:05.875523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.875422    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a19dd1e-502f-4d23-8666-a05938cef188" containerName="revoke-keys"
Oct 29 23:32:05.875523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.875509    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39c2cc6-0db7-4b32-9786-f5b911fa6e52" containerName="revoke-keys"
Oct 29 23:32:05.875523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.875521    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a19dd1e-502f-4d23-8666-a05938cef188" containerName="revoke-keys"
Oct 29 23:32:05.879442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.879425    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:05.900572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.900545    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"]
Oct 29 23:32:05.909931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.909905    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:32:05.910081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.909976    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:05.910081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.910022    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:32:05.911587 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.911569    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:32:05.960335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.960298    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:05.960505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.960353    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvg7\" (UniqueName: \"kubernetes.io/projected/dff80c93-47a4-4563-9af9-93cf16356b51-kube-api-access-7qvg7\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:05.960505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:05.960447    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.062046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.062011    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.062202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.062073    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvg7\" (UniqueName: \"kubernetes.io/projected/dff80c93-47a4-4563-9af9-93cf16356b51-kube-api-access-7qvg7\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.062202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.062135    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.062712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.062693    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.064452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.064428    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.091316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.091285    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvg7\" (UniqueName: \"kubernetes.io/projected/dff80c93-47a4-4563-9af9-93cf16356b51-kube-api-access-7qvg7\") pod \"maas-api-revoke-sub-fc838e28-nnfnj\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.193478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.193442    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:06.496395 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:06.496362    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff80c93_47a4_4563_9af9_93cf16356b51.slice/crio-cb0a20a0d83f76b6fc20bca5c0c51b1c3c2be65e801f9c5f7a4608d7ac9570fa WatchSource:0}: Error finding container cb0a20a0d83f76b6fc20bca5c0c51b1c3c2be65e801f9c5f7a4608d7ac9570fa: Status 404 returned error can't find the container with id cb0a20a0d83f76b6fc20bca5c0c51b1c3c2be65e801f9c5f7a4608d7ac9570fa
Oct 29 23:32:06.496574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:06.496551    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"]
Oct 29 23:32:07.370515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:07.370486    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj" event={"ID":"dff80c93-47a4-4563-9af9-93cf16356b51","Type":"ContainerStarted","Data":"a829b8351cb57c28f96de67267df2121553b1da75295aed68ae118f19e48b6e5"}
Oct 29 23:32:07.370862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:07.370535    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj" event={"ID":"dff80c93-47a4-4563-9af9-93cf16356b51","Type":"ContainerStarted","Data":"cb0a20a0d83f76b6fc20bca5c0c51b1c3c2be65e801f9c5f7a4608d7ac9570fa"}
Oct 29 23:32:08.380079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.380042    6203 generic.go:356] "Generic (PLEG): container finished" podID="dff80c93-47a4-4563-9af9-93cf16356b51" containerID="a829b8351cb57c28f96de67267df2121553b1da75295aed68ae118f19e48b6e5" exitCode=0
Oct 29 23:32:08.380638 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.380121    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj" event={"ID":"dff80c93-47a4-4563-9af9-93cf16356b51","Type":"ContainerDied","Data":"a829b8351cb57c28f96de67267df2121553b1da75295aed68ae118f19e48b6e5"}
Oct 29 23:32:08.556382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.556353    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:08.598516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.598479    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff80c93-47a4-4563-9af9-93cf16356b51-kube-api-access-7qvg7\" (UniqueName: \"kubernetes.io/projected/dff80c93-47a4-4563-9af9-93cf16356b51-kube-api-access-7qvg7\") pod \"dff80c93-47a4-4563-9af9-93cf16356b51\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") "
Oct 29 23:32:08.598691 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.598585    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-openshift-ca\") pod \"dff80c93-47a4-4563-9af9-93cf16356b51\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") "
Oct 29 23:32:08.598807 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.598790    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-xks-ca\") pod \"dff80c93-47a4-4563-9af9-93cf16356b51\" (UID: \"dff80c93-47a4-4563-9af9-93cf16356b51\") "
Oct 29 23:32:08.599054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.599018    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-openshift-ca" pod "dff80c93-47a4-4563-9af9-93cf16356b51" (UID: "dff80c93-47a4-4563-9af9-93cf16356b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:08.599537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.599499    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.601022 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.600990    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-xks-ca" pod "dff80c93-47a4-4563-9af9-93cf16356b51" (UID: "dff80c93-47a4-4563-9af9-93cf16356b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:08.601241 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.601213    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff80c93-47a4-4563-9af9-93cf16356b51-kube-api-access-7qvg7" pod "dff80c93-47a4-4563-9af9-93cf16356b51" (UID: "dff80c93-47a4-4563-9af9-93cf16356b51"). InnerVolumeSpecName "kube-api-access-7qvg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:08.723909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.721226    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff80c93-47a4-4563-9af9-93cf16356b51-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:08.723909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:08.721269    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvg7\" (UniqueName: \"kubernetes.io/projected/dff80c93-47a4-4563-9af9-93cf16356b51-kube-api-access-7qvg7\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:09.410903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:09.410323    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj" event={"ID":"dff80c93-47a4-4563-9af9-93cf16356b51","Type":"ContainerDied","Data":"cb0a20a0d83f76b6fc20bca5c0c51b1c3c2be65e801f9c5f7a4608d7ac9570fa"}
Oct 29 23:32:09.410903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:09.410394    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb0a20a0d83f76b6fc20bca5c0c51b1c3c2be65e801f9c5f7a4608d7ac9570fa"
Oct 29 23:32:09.410903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:09.410473    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"
Oct 29 23:32:10.150953 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:32:10.150761    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:32:10.452014 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:10.451964    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:10.504901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:10.503991    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Oct 29 23:32:10.607063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:10.607012    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:10.773960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:10.762996    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:10.828904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:10.826317    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:10.908376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:10.908324    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:10.932907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:10.931415    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:11.115963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.114979    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:11.119933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.116281    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:11.221902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.220978    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:11.481918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.478984    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:11.481918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.481031    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:32:11.783962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.778593    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:32:11.783962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.780161    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:32:11.833921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.829970    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:11.833921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.831813    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:11.916904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.914170    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:11.924904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.919464    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:32:11.955914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:11.939209    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 23:32:12.477954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.204018    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:12.587942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.209965    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:32:12.802949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.802820    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:12.809973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.809931    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:32:12.933903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.925980    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:12.958903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.934360    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj" podUID="5c660259-6736-4175-a257-4e65889e0761" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Oct 29 23:32:12.958903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.942921    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 23:32:13.011901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:12.987025    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:13.304433 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:32:13.301901    6203 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.253s"
Oct 29 23:32:13.304433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.302034    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:32:13.304433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.302075    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:32:13.304433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.302114    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:32:13.304433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.302156    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:32:13.493908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.475982    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:13.527967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.527919    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:13.580954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.580259    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:13.809570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.809538    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:32:13.810591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:13.810556    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:32:14.049023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.048979    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:32:14.050426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.050385    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 23:32:14.115905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.065380    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:14.117056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.116927    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:14.217926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.217864    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:14.220550 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.220526    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:32:14.220793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.220764    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:32:14.221000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.220971    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:14.269485 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.269441    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:14.351419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.351328    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:32:14.440478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.439407    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:32:14.455898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.446735    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:32:14.479284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:14.479197    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:32:16.377737 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.377704    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"]
Oct 29 23:32:16.378639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.378618    6203 cpu_manager.go:414] "containerMap: removing container" podUID="dff80c93-47a4-4563-9af9-93cf16356b51" containerName="revoke-keys"
Oct 29 23:32:16.378700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.378641    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff80c93-47a4-4563-9af9-93cf16356b51" containerName="revoke-keys"
Oct 29 23:32:16.378842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.378830    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff80c93-47a4-4563-9af9-93cf16356b51" containerName="revoke-keys"
Oct 29 23:32:16.384249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.384223    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.404307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.404271    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"]
Oct 29 23:32:16.412034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.412008    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:16.412218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.412069    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:32:16.412218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.412103    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:32:16.417199 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.417173    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:32:16.512815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.512783    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.512996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.512864    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.512996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.512956    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6m76\" (UniqueName: \"kubernetes.io/projected/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-kube-api-access-f6m76\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.613825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.613790    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.614070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.613874    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.614070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.613991    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6m76\" (UniqueName: \"kubernetes.io/projected/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-kube-api-access-f6m76\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.614557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.614534    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.616238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.616218    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.640201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.640134    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6m76\" (UniqueName: \"kubernetes.io/projected/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-kube-api-access-f6m76\") pod \"maas-api-revoke-sub-68ddf24a-hl9bs\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:16.702365 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:16.702333    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:17.009201 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:17.009163    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff3ed99_48d3_4d2d_84fa_e23ffa720a08.slice/crio-a98c351d1caf7857f5aee92429e724c1635f20ae9a9c5560f672746b79149d38 WatchSource:0}: Error finding container a98c351d1caf7857f5aee92429e724c1635f20ae9a9c5560f672746b79149d38: Status 404 returned error can't find the container with id a98c351d1caf7857f5aee92429e724c1635f20ae9a9c5560f672746b79149d38
Oct 29 23:32:17.010078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:17.010053    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"]
Oct 29 23:32:17.357775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:17.357737    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs" event={"ID":"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08","Type":"ContainerStarted","Data":"73dca94782d7e4b18d63121f610b80a017e348f41ce851150f319f6bae588d74"}
Oct 29 23:32:17.357940 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:17.357799    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs" event={"ID":"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08","Type":"ContainerStarted","Data":"a98c351d1caf7857f5aee92429e724c1635f20ae9a9c5560f672746b79149d38"}
Oct 29 23:32:17.407163 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:17.407100    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs" podStartSLOduration=1.407082471 podStartE2EDuration="1.407082471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:17.404633883 +0000 UTC m=+1115.508894269" watchObservedRunningTime="2026-10-29 23:32:17.407082471 +0000 UTC m=+1115.511342882"
Oct 29 23:32:18.359275 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.359193    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"]
Oct 29 23:32:18.363355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.363333    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.366609 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.366575    6203 generic.go:356] "Generic (PLEG): container finished" podID="9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" containerID="73dca94782d7e4b18d63121f610b80a017e348f41ce851150f319f6bae588d74" exitCode=0
Oct 29 23:32:18.366728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.366652    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs" event={"ID":"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08","Type":"ContainerDied","Data":"73dca94782d7e4b18d63121f610b80a017e348f41ce851150f319f6bae588d74"}
Oct 29 23:32:18.384500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.384470    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"]
Oct 29 23:32:18.541892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.541828    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.542281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.542103    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.542359 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.542318    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjhb\" (UniqueName: \"kubernetes.io/projected/7bf91e33-927c-4888-a6b8-1c5875e673a6-kube-api-access-bgjhb\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.643385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.643297    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjhb\" (UniqueName: \"kubernetes.io/projected/7bf91e33-927c-4888-a6b8-1c5875e673a6-kube-api-access-bgjhb\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.643525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.643428    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.643584 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.643574    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.644083 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.644057    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.645976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.645930    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.669457 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.669424    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjhb\" (UniqueName: \"kubernetes.io/projected/7bf91e33-927c-4888-a6b8-1c5875e673a6-kube-api-access-bgjhb\") pod \"maas-api-revoke-sub-94a06a50-t7q6s\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.679386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.679358    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:18.998897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:18.998836    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"]
Oct 29 23:32:18.999951 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:18.999872    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf91e33_927c_4888_a6b8_1c5875e673a6.slice/crio-06f63d9f24da7f37c533f8038c7a1cb613b721b5ff6cd7ee98f73710021be3ae WatchSource:0}: Error finding container 06f63d9f24da7f37c533f8038c7a1cb613b721b5ff6cd7ee98f73710021be3ae: Status 404 returned error can't find the container with id 06f63d9f24da7f37c533f8038c7a1cb613b721b5ff6cd7ee98f73710021be3ae
Oct 29 23:32:19.375924 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.375873    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s" event={"ID":"7bf91e33-927c-4888-a6b8-1c5875e673a6","Type":"ContainerStarted","Data":"650217440e833c99f5d11d63b665be2e4d2c0b727213d461fd9eb78c6233eef6"}
Oct 29 23:32:19.376107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.375948    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s" event={"ID":"7bf91e33-927c-4888-a6b8-1c5875e673a6","Type":"ContainerStarted","Data":"06f63d9f24da7f37c533f8038c7a1cb613b721b5ff6cd7ee98f73710021be3ae"}
Oct 29 23:32:19.423784 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.423719    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s" podStartSLOduration=1.423704956 podStartE2EDuration="1.423704956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:19.421555758 +0000 UTC m=+1117.525816144" watchObservedRunningTime="2026-10-29 23:32:19.423704956 +0000 UTC m=+1117.527965341"
Oct 29 23:32:19.517294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.517268    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:19.553855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.553822    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-xks-ca\") pod \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") "
Oct 29 23:32:19.554295 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.554034    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-kube-api-access-f6m76\" (UniqueName: \"kubernetes.io/projected/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-kube-api-access-f6m76\") pod \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") "
Oct 29 23:32:19.554295 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.554082    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-openshift-ca\") pod \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\" (UID: \"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08\") "
Oct 29 23:32:19.554542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.554516    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-openshift-ca" pod "9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" (UID: "9ff3ed99-48d3-4d2d-84fa-e23ffa720a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:19.554848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.554829    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:19.556072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.556046    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-xks-ca" pod "9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" (UID: "9ff3ed99-48d3-4d2d-84fa-e23ffa720a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:19.556190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.556171    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-kube-api-access-f6m76" pod "9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" (UID: "9ff3ed99-48d3-4d2d-84fa-e23ffa720a08"). InnerVolumeSpecName "kube-api-access-f6m76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:19.655641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.655566    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6m76\" (UniqueName: \"kubernetes.io/projected/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-kube-api-access-f6m76\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:19.655641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.655593    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:19.813127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.813095    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"]
Oct 29 23:32:19.813720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.813705    6203 cpu_manager.go:414] "containerMap: removing container" podUID="9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" containerName="revoke-keys"
Oct 29 23:32:19.813780 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.813720    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" containerName="revoke-keys"
Oct 29 23:32:19.813837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.813827    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" containerName="revoke-keys"
Oct 29 23:32:19.817907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.817890    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.837147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.837123    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ae1364\"/\"distinct-w2-ae1364-kserve-self-signed-certs\""
Oct 29 23:32:19.848410 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.848351    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"]
Oct 29 23:32:19.857677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.857644    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-model-cache\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.857819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.857705    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcn9x\" (UniqueName: \"kubernetes.io/projected/087a38a6-8300-4b22-8966-e3d0e18fb2fb-kube-api-access-kcn9x\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.857901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.857828    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-dshm\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.857979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.857957    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tmp-dir\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.858037 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.858026    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tls-certs\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.858143 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.858130    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-home\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.959364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.959335    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-dshm\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.959532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.959392    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tmp-dir\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.959532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.959472    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tls-certs\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.959645 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.959602    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-home\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.959740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.959724    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-model-cache\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.959830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.959745    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tmp-dir\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.959830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.959807    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcn9x\" (UniqueName: \"kubernetes.io/projected/087a38a6-8300-4b22-8966-e3d0e18fb2fb-kube-api-access-kcn9x\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.960042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.960015    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-home\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.960134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.960046    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-model-cache\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.961974 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.961948    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-dshm\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.962091 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.962070    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tls-certs\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:19.993287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:19.993248    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcn9x\" (UniqueName: \"kubernetes.io/projected/087a38a6-8300-4b22-8966-e3d0e18fb2fb-kube-api-access-kcn9x\") pod \"distinct-w2-ae1364-kserve-bf5fcb855-zcxvs\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") " pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:20.131269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.131232    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:20.389842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.389798    6203 generic.go:356] "Generic (PLEG): container finished" podID="7bf91e33-927c-4888-a6b8-1c5875e673a6" containerID="650217440e833c99f5d11d63b665be2e4d2c0b727213d461fd9eb78c6233eef6" exitCode=0
Oct 29 23:32:20.390059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.389902    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s" event={"ID":"7bf91e33-927c-4888-a6b8-1c5875e673a6","Type":"ContainerDied","Data":"650217440e833c99f5d11d63b665be2e4d2c0b727213d461fd9eb78c6233eef6"}
Oct 29 23:32:20.391823 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.391794    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs" event={"ID":"9ff3ed99-48d3-4d2d-84fa-e23ffa720a08","Type":"ContainerDied","Data":"a98c351d1caf7857f5aee92429e724c1635f20ae9a9c5560f672746b79149d38"}
Oct 29 23:32:20.392039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.391838    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a98c351d1caf7857f5aee92429e724c1635f20ae9a9c5560f672746b79149d38"
Oct 29 23:32:20.392039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.391918    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"
Oct 29 23:32:20.450550 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:20.450486    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod087a38a6_8300_4b22_8966_e3d0e18fb2fb.slice/crio-6478849e347699aa3077ab630cd26eb4af737a801e10517f5a65286e65a7e1dd WatchSource:0}: Error finding container 6478849e347699aa3077ab630cd26eb4af737a801e10517f5a65286e65a7e1dd: Status 404 returned error can't find the container with id 6478849e347699aa3077ab630cd26eb4af737a801e10517f5a65286e65a7e1dd
Oct 29 23:32:20.455241 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.455215    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"]
Oct 29 23:32:20.907868 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.907778    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"]
Oct 29 23:32:20.911965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.911940    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:20.936047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.936015    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"]
Oct 29 23:32:20.975298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.975265    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrxxv\" (UniqueName: \"kubernetes.io/projected/61de0dcd-e20f-4365-91a8-e4bd91b674a2-kube-api-access-qrxxv\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:20.975464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.975397    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:20.975578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:20.975560    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.076571 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.076537    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrxxv\" (UniqueName: \"kubernetes.io/projected/61de0dcd-e20f-4365-91a8-e4bd91b674a2-kube-api-access-qrxxv\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.076744 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.076610    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.076744 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.076717    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.077384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.077357    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.079033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.079012    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.102212 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.102191    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrxxv\" (UniqueName: \"kubernetes.io/projected/61de0dcd-e20f-4365-91a8-e4bd91b674a2-kube-api-access-qrxxv\") pod \"maas-api-revoke-sub-106efa1b-jbrcx\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.226294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.226263    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:21.406066 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.406023    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" event={"ID":"087a38a6-8300-4b22-8966-e3d0e18fb2fb","Type":"ContainerStarted","Data":"5632ad55083ef383e02aa9cce91176927f41d511516d6991adc5687f23d29619"}
Oct 29 23:32:21.406273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.406080    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" event={"ID":"087a38a6-8300-4b22-8966-e3d0e18fb2fb","Type":"ContainerStarted","Data":"6478849e347699aa3077ab630cd26eb4af737a801e10517f5a65286e65a7e1dd"}
Oct 29 23:32:21.456120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.456061    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podStartSLOduration=2.45604659 podStartE2EDuration="2.45604659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:21.453180846 +0000 UTC m=+1119.557441233" watchObservedRunningTime="2026-10-29 23:32:21.45604659 +0000 UTC m=+1119.560306975"
Oct 29 23:32:21.530926 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:21.530797    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61de0dcd_e20f_4365_91a8_e4bd91b674a2.slice/crio-cf2a65a86d8249638506986b5956c5adaa468bc2ed7e4cbe4bb19bd12c9f84fc WatchSource:0}: Error finding container cf2a65a86d8249638506986b5956c5adaa468bc2ed7e4cbe4bb19bd12c9f84fc: Status 404 returned error can't find the container with id cf2a65a86d8249638506986b5956c5adaa468bc2ed7e4cbe4bb19bd12c9f84fc
Oct 29 23:32:21.530926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.530814    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"]
Oct 29 23:32:21.541476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.541456    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:21.585900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.584331    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-openshift-ca\") pod \"7bf91e33-927c-4888-a6b8-1c5875e673a6\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") "
Oct 29 23:32:21.585900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.584730    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-openshift-ca" pod "7bf91e33-927c-4888-a6b8-1c5875e673a6" (UID: "7bf91e33-927c-4888-a6b8-1c5875e673a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:21.585900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.585026    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf91e33-927c-4888-a6b8-1c5875e673a6-kube-api-access-bgjhb\" (UniqueName: \"kubernetes.io/projected/7bf91e33-927c-4888-a6b8-1c5875e673a6-kube-api-access-bgjhb\") pod \"7bf91e33-927c-4888-a6b8-1c5875e673a6\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") "
Oct 29 23:32:21.585900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.585172    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-xks-ca\") pod \"7bf91e33-927c-4888-a6b8-1c5875e673a6\" (UID: \"7bf91e33-927c-4888-a6b8-1c5875e673a6\") "
Oct 29 23:32:21.585900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.585725    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:21.592261 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.590340    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf91e33-927c-4888-a6b8-1c5875e673a6-kube-api-access-bgjhb" pod "7bf91e33-927c-4888-a6b8-1c5875e673a6" (UID: "7bf91e33-927c-4888-a6b8-1c5875e673a6"). InnerVolumeSpecName "kube-api-access-bgjhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:21.592261 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.591144    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-xks-ca" pod "7bf91e33-927c-4888-a6b8-1c5875e673a6" (UID: "7bf91e33-927c-4888-a6b8-1c5875e673a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:21.687128 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.687097    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf91e33-927c-4888-a6b8-1c5875e673a6-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:21.687128 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:21.687125    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjhb\" (UniqueName: \"kubernetes.io/projected/7bf91e33-927c-4888-a6b8-1c5875e673a6-kube-api-access-bgjhb\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:22.131994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:22.131908    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:22.414448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:22.414360    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"
Oct 29 23:32:22.414448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:22.414396    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s" event={"ID":"7bf91e33-927c-4888-a6b8-1c5875e673a6","Type":"ContainerDied","Data":"06f63d9f24da7f37c533f8038c7a1cb613b721b5ff6cd7ee98f73710021be3ae"}
Oct 29 23:32:22.414678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:22.414452    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f63d9f24da7f37c533f8038c7a1cb613b721b5ff6cd7ee98f73710021be3ae"
Oct 29 23:32:22.420805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:22.420771    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx" event={"ID":"61de0dcd-e20f-4365-91a8-e4bd91b674a2","Type":"ContainerStarted","Data":"da46d465b60ce6c6c25ceb0ce1d850ab540ddedb72f86d5cf72da91da9f6113e"}
Oct 29 23:32:22.421199 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:22.420821    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx" event={"ID":"61de0dcd-e20f-4365-91a8-e4bd91b674a2","Type":"ContainerStarted","Data":"cf2a65a86d8249638506986b5956c5adaa468bc2ed7e4cbe4bb19bd12c9f84fc"}
Oct 29 23:32:22.489861 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:22.489804    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx" podStartSLOduration=2.489790598 podStartE2EDuration="2.489790598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:22.488234027 +0000 UTC m=+1120.592494410" watchObservedRunningTime="2026-10-29 23:32:22.489790598 +0000 UTC m=+1120.594051009"
Oct 29 23:32:23.430744 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:23.430708    6203 generic.go:356] "Generic (PLEG): container finished" podID="61de0dcd-e20f-4365-91a8-e4bd91b674a2" containerID="da46d465b60ce6c6c25ceb0ce1d850ab540ddedb72f86d5cf72da91da9f6113e" exitCode=0
Oct 29 23:32:23.431192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:23.430785    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx" event={"ID":"61de0dcd-e20f-4365-91a8-e4bd91b674a2","Type":"ContainerDied","Data":"da46d465b60ce6c6c25ceb0ce1d850ab540ddedb72f86d5cf72da91da9f6113e"}
Oct 29 23:32:24.473661 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.473614    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"]
Oct 29 23:32:24.474478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.474455    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7bf91e33-927c-4888-a6b8-1c5875e673a6" containerName="revoke-keys"
Oct 29 23:32:24.474581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.474479    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf91e33-927c-4888-a6b8-1c5875e673a6" containerName="revoke-keys"
Oct 29 23:32:24.474684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.474670    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf91e33-927c-4888-a6b8-1c5875e673a6" containerName="revoke-keys"
Oct 29 23:32:24.478542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.478518    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.571915 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.571838    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"]
Oct 29 23:32:24.574898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.574800    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.575264 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.575191    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnpp\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-kube-api-access-wvnpp\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.576064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.575390    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.576478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.576353    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7f61d-89b9-4358-84fa-75e6d6201317-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.576717 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.576604    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.576717 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.576670    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.577079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.576927    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.577287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.577263    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.577381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.577365    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.580087 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.578225    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.581828 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.581808    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"]
Oct 29 23:32:24.589921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.589830    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"]
Oct 29 23:32:24.638624 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.638594    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:24.678470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678437    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.678470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678479    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29850449-f8b7-4c0f-9240-46fc86789ff2-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.678693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678524    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.678693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678608    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.678693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678650    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.678824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678707    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.678824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678776    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.678928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678840    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.678928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678908    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.679013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678944    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.679013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.678981    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnpp\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-kube-api-access-wvnpp\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.679141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.679017    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.679141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.679063    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.679141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.679111    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7f61d-89b9-4358-84fa-75e6d6201317-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.679141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.679119    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.679336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.679140    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9d2c\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-kube-api-access-v9d2c\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.679688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.679660    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.679962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.679943    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7f61d-89b9-4358-84fa-75e6d6201317-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.680065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.680026    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.680310 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.680291    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.680417 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.680339    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.681146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.680752    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.681146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.681088    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.682765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.682739    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.682895 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.682814    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.706206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.706133    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnpp\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-kube-api-access-wvnpp\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.707421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.707380    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.781427 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.781396    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61de0dcd-e20f-4365-91a8-e4bd91b674a2-kube-api-access-qrxxv\" (UniqueName: \"kubernetes.io/projected/61de0dcd-e20f-4365-91a8-e4bd91b674a2-kube-api-access-qrxxv\") pod \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") "
Oct 29 23:32:24.781579 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.781565    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-xks-ca\") pod \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") "
Oct 29 23:32:24.781636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.781597    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-openshift-ca\") pod \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\" (UID: \"61de0dcd-e20f-4365-91a8-e4bd91b674a2\") "
Oct 29 23:32:24.781810 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.781790    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.781918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.781844    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.781982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.781950    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.781989    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782075    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-openshift-ca" pod "61de0dcd-e20f-4365-91a8-e4bd91b674a2" (UID: "61de0dcd-e20f-4365-91a8-e4bd91b674a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:24.782277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782253    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782349 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782329    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782358    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782488    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9d2c\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-kube-api-access-v9d2c\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782601    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782629    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29850449-f8b7-4c0f-9240-46fc86789ff2-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782660    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782784 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782707    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.782837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.782805    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:24.783102 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.783079    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.783287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.783267    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29850449-f8b7-4c0f-9240-46fc86789ff2-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.783935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.783913    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-xks-ca" pod "61de0dcd-e20f-4365-91a8-e4bd91b674a2" (UID: "61de0dcd-e20f-4365-91a8-e4bd91b674a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:24.784008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.783942    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61de0dcd-e20f-4365-91a8-e4bd91b674a2-kube-api-access-qrxxv" pod "61de0dcd-e20f-4365-91a8-e4bd91b674a2" (UID: "61de0dcd-e20f-4365-91a8-e4bd91b674a2"). InnerVolumeSpecName "kube-api-access-qrxxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:24.784728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.784711    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.785056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.785040    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.794219 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.794200    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:24.810860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.810822    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.811003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.810909    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9d2c\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-kube-api-access-v9d2c\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:24.884929 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.884673    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrxxv\" (UniqueName: \"kubernetes.io/projected/61de0dcd-e20f-4365-91a8-e4bd91b674a2-kube-api-access-qrxxv\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:24.884929 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.884711    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61de0dcd-e20f-4365-91a8-e4bd91b674a2-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:24.905686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:24.905652    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:25.118164 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:25.118135    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a7f61d_89b9_4358_84fa_75e6d6201317.slice/crio-6394e606efdfbaa7e95eb0ee3ea6f4812663de46a73c296f1e7ee517e5a8a7d1 WatchSource:0}: Error finding container 6394e606efdfbaa7e95eb0ee3ea6f4812663de46a73c296f1e7ee517e5a8a7d1: Status 404 returned error can't find the container with id 6394e606efdfbaa7e95eb0ee3ea6f4812663de46a73c296f1e7ee517e5a8a7d1
Oct 29 23:32:25.119268 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.119228    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"]
Oct 29 23:32:25.120735 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.120700    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:25.120823 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.120772    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:25.120823 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.120804    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:25.217483 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:25.217454    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29850449_f8b7_4c0f_9240_46fc86789ff2.slice/crio-f17d2c56ba1d999848a4dc229ddf273889fce44afcef4863c4e6ec49b5238631 WatchSource:0}: Error finding container f17d2c56ba1d999848a4dc229ddf273889fce44afcef4863c4e6ec49b5238631: Status 404 returned error can't find the container with id f17d2c56ba1d999848a4dc229ddf273889fce44afcef4863c4e6ec49b5238631
Oct 29 23:32:25.219050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.219016    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"]
Oct 29 23:32:25.220730 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.220681    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:25.220844 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.220771    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:25.220844 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.220811    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:25.452968 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.452930    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" event={"ID":"29850449-f8b7-4c0f-9240-46fc86789ff2","Type":"ContainerStarted","Data":"a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935"}
Oct 29 23:32:25.453157 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.452991    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" event={"ID":"29850449-f8b7-4c0f-9240-46fc86789ff2","Type":"ContainerStarted","Data":"f17d2c56ba1d999848a4dc229ddf273889fce44afcef4863c4e6ec49b5238631"}
Oct 29 23:32:25.456130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.456088    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" event={"ID":"05a7f61d-89b9-4358-84fa-75e6d6201317","Type":"ContainerStarted","Data":"0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42"}
Oct 29 23:32:25.456293 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.456152    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" event={"ID":"05a7f61d-89b9-4358-84fa-75e6d6201317","Type":"ContainerStarted","Data":"6394e606efdfbaa7e95eb0ee3ea6f4812663de46a73c296f1e7ee517e5a8a7d1"}
Oct 29 23:32:25.458676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.458648    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx" event={"ID":"61de0dcd-e20f-4365-91a8-e4bd91b674a2","Type":"ContainerDied","Data":"cf2a65a86d8249638506986b5956c5adaa468bc2ed7e4cbe4bb19bd12c9f84fc"}
Oct 29 23:32:25.458846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.458834    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf2a65a86d8249638506986b5956c5adaa468bc2ed7e4cbe4bb19bd12c9f84fc"
Oct 29 23:32:25.458946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.458692    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"
Oct 29 23:32:25.505033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.504971    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" podStartSLOduration=1.504952133 podStartE2EDuration="1.504952133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:25.504261161 +0000 UTC m=+1123.608521547" watchObservedRunningTime="2026-10-29 23:32:25.504952133 +0000 UTC m=+1123.609212520"
Oct 29 23:32:25.576482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.576420    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podStartSLOduration=1.576401173 podStartE2EDuration="1.576401173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:25.572038161 +0000 UTC m=+1123.676298559" watchObservedRunningTime="2026-10-29 23:32:25.576401173 +0000 UTC m=+1123.680661559"
Oct 29 23:32:25.795200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.795119    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:25.906293 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:25.906264    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:26.796989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:26.796957    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 29 23:32:26.797469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:26.797441    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 29 23:32:26.908477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:26.908442    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 29 23:32:26.908831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:26.908807    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 29 23:32:27.795452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:27.795411    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 29 23:32:27.795678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:27.795473    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 29 23:32:27.906710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:27.906676    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 29 23:32:27.907130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:27.906734    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 29 23:32:28.793376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.793311    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"]
Oct 29 23:32:28.794026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.794011    6203 cpu_manager.go:414] "containerMap: removing container" podUID="61de0dcd-e20f-4365-91a8-e4bd91b674a2" containerName="revoke-keys"
Oct 29 23:32:28.794097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.794027    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61de0dcd-e20f-4365-91a8-e4bd91b674a2" containerName="revoke-keys"
Oct 29 23:32:28.794133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.794120    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="61de0dcd-e20f-4365-91a8-e4bd91b674a2" containerName="revoke-keys"
Oct 29 23:32:28.795221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.795192    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 29 23:32:28.795357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.795305    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 29 23:32:28.799246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.799228    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:28.822903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.822013    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"]
Oct 29 23:32:28.866904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.862489    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:32:28.866904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.862715    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:32:28.866904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.863102    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:32:28.866904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.863177    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:28.889423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.888668    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:28.907167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.906646    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 29 23:32:28.907167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.906712    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 29 23:32:28.996117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.993121    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:28.996117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.993343    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:28.996117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.993452    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbxx\" (UniqueName: \"kubernetes.io/projected/19fb5de7-9a1e-4f50-8abd-33a64a830065-kube-api-access-jpbxx\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:28.996117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:28.993834    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:29.096401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.096297    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:29.096574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.096460    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbxx\" (UniqueName: \"kubernetes.io/projected/19fb5de7-9a1e-4f50-8abd-33a64a830065-kube-api-access-jpbxx\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:29.107279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.107231    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:29.172946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.158635    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbxx\" (UniqueName: \"kubernetes.io/projected/19fb5de7-9a1e-4f50-8abd-33a64a830065-kube-api-access-jpbxx\") pod \"maas-api-revoke-sub-00c79458-j2ssb\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:29.414716 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.414634    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:29.788083 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:29.788041    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fb5de7_9a1e_4f50_8abd_33a64a830065.slice/crio-a2c3cd61d1ac6ee250c9e18d6be2a3e3582dbf283aa11d91c7eab4f76332fa62 WatchSource:0}: Error finding container a2c3cd61d1ac6ee250c9e18d6be2a3e3582dbf283aa11d91c7eab4f76332fa62: Status 404 returned error can't find the container with id a2c3cd61d1ac6ee250c9e18d6be2a3e3582dbf283aa11d91c7eab4f76332fa62
Oct 29 23:32:29.788351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.788329    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"]
Oct 29 23:32:29.794895 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.794863    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 29 23:32:29.794976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.794919    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 29 23:32:29.906031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.905992    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 29 23:32:29.906231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:29.906055    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 29 23:32:30.132312 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.132278    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:30.192357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.192325    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:30.518364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.518322    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb" event={"ID":"19fb5de7-9a1e-4f50-8abd-33a64a830065","Type":"ContainerStarted","Data":"e95b97c9dd0972d1c27b4b5211fcd86bc03485ae91c47f823963de731650634e"}
Oct 29 23:32:30.518364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.518374    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb" event={"ID":"19fb5de7-9a1e-4f50-8abd-33a64a830065","Type":"ContainerStarted","Data":"a2c3cd61d1ac6ee250c9e18d6be2a3e3582dbf283aa11d91c7eab4f76332fa62"}
Oct 29 23:32:30.589901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.589717    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:30.657902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.649957    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb" podStartSLOduration=2.6499341789999997 podStartE2EDuration="2.649934179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:30.584515458 +0000 UTC m=+1128.688775845" watchObservedRunningTime="2026-10-29 23:32:30.649934179 +0000 UTC m=+1128.754194566"
Oct 29 23:32:30.794630 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.794544    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 29 23:32:30.794630 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.794589    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 29 23:32:30.907061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.907021    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 29 23:32:30.907218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:30.907082    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 29 23:32:31.529623 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:31.529588    6203 generic.go:356] "Generic (PLEG): container finished" podID="19fb5de7-9a1e-4f50-8abd-33a64a830065" containerID="e95b97c9dd0972d1c27b4b5211fcd86bc03485ae91c47f823963de731650634e" exitCode=0
Oct 29 23:32:31.530108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:31.529677    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb" event={"ID":"19fb5de7-9a1e-4f50-8abd-33a64a830065","Type":"ContainerDied","Data":"e95b97c9dd0972d1c27b4b5211fcd86bc03485ae91c47f823963de731650634e"}
Oct 29 23:32:31.794988 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:31.794903    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 29 23:32:31.794988 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:31.794970    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 29 23:32:32.080248 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.080124    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:32.080775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.080752    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:32.081745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.081720    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:32:32.617180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.616254    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"]
Oct 29 23:32:32.621589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.621559    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.645004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.644973    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"]
Oct 29 23:32:32.650812 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.650776    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.650986 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.650837    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvwc\" (UniqueName: \"kubernetes.io/projected/8069f4d6-da2c-4306-8db6-b7fe71262feb-kube-api-access-rhvwc\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.650986 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.650971    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.710288 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.710264    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:32.752062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.752029    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fb5de7-9a1e-4f50-8abd-33a64a830065-kube-api-access-jpbxx\" (UniqueName: \"kubernetes.io/projected/19fb5de7-9a1e-4f50-8abd-33a64a830065-kube-api-access-jpbxx\") pod \"19fb5de7-9a1e-4f50-8abd-33a64a830065\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") "
Oct 29 23:32:32.752252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.752196    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-openshift-ca\") pod \"19fb5de7-9a1e-4f50-8abd-33a64a830065\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") "
Oct 29 23:32:32.752312 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.752288    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-xks-ca\") pod \"19fb5de7-9a1e-4f50-8abd-33a64a830065\" (UID: \"19fb5de7-9a1e-4f50-8abd-33a64a830065\") "
Oct 29 23:32:32.752586 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.752557    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-openshift-ca" pod "19fb5de7-9a1e-4f50-8abd-33a64a830065" (UID: "19fb5de7-9a1e-4f50-8abd-33a64a830065"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:32.752820 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.752801    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.752950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.752845    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvwc\" (UniqueName: \"kubernetes.io/projected/8069f4d6-da2c-4306-8db6-b7fe71262feb-kube-api-access-rhvwc\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.752950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.752940    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.753177 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.753120    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:32.753549 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.753521    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.754708 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.754677    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-xks-ca" pod "19fb5de7-9a1e-4f50-8abd-33a64a830065" (UID: "19fb5de7-9a1e-4f50-8abd-33a64a830065"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:32.754900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.754856    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fb5de7-9a1e-4f50-8abd-33a64a830065-kube-api-access-jpbxx" pod "19fb5de7-9a1e-4f50-8abd-33a64a830065" (UID: "19fb5de7-9a1e-4f50-8abd-33a64a830065"). InnerVolumeSpecName "kube-api-access-jpbxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:32.755682 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.755657    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.780098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.780040    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvwc\" (UniqueName: \"kubernetes.io/projected/8069f4d6-da2c-4306-8db6-b7fe71262feb-kube-api-access-rhvwc\") pod \"maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:32.800247 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.800185    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:32.800742 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.800726    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:32.802176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.802153    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:32:32.854530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.854497    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbxx\" (UniqueName: \"kubernetes.io/projected/19fb5de7-9a1e-4f50-8abd-33a64a830065-kube-api-access-jpbxx\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:32.854530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.854526    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fb5de7-9a1e-4f50-8abd-33a64a830065-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:32.940061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:32.940023    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:33.264493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:33.264465    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"]
Oct 29 23:32:33.265082 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:33.265051    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8069f4d6_da2c_4306_8db6_b7fe71262feb.slice/crio-0c5d551d7d82e6213bd051a9750cc77f211ab4ac249d46e714be60f5d352985b WatchSource:0}: Error finding container 0c5d551d7d82e6213bd051a9750cc77f211ab4ac249d46e714be60f5d352985b: Status 404 returned error can't find the container with id 0c5d551d7d82e6213bd051a9750cc77f211ab4ac249d46e714be60f5d352985b
Oct 29 23:32:33.547903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:33.547851    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb" event={"ID":"19fb5de7-9a1e-4f50-8abd-33a64a830065","Type":"ContainerDied","Data":"a2c3cd61d1ac6ee250c9e18d6be2a3e3582dbf283aa11d91c7eab4f76332fa62"}
Oct 29 23:32:33.548036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:33.547913    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2c3cd61d1ac6ee250c9e18d6be2a3e3582dbf283aa11d91c7eab4f76332fa62"
Oct 29 23:32:33.548036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:33.547861    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"
Oct 29 23:32:33.549779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:33.549751    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x" event={"ID":"8069f4d6-da2c-4306-8db6-b7fe71262feb","Type":"ContainerStarted","Data":"0c5d551d7d82e6213bd051a9750cc77f211ab4ac249d46e714be60f5d352985b"}
Oct 29 23:32:33.598758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:33.598706    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x" podStartSLOduration=1.598693559 podStartE2EDuration="1.598693559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:33.59745147 +0000 UTC m=+1131.701711857" watchObservedRunningTime="2026-10-29 23:32:33.598693559 +0000 UTC m=+1131.702953944"
Oct 29 23:32:34.571033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:34.570985    6203 generic.go:356] "Generic (PLEG): container finished" podID="8069f4d6-da2c-4306-8db6-b7fe71262feb" containerID="8c063906ed311ff96cb517c3a787d5acd0dc291cd84c2f03b0568a0354806b67" exitCode=0
Oct 29 23:32:34.571444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:34.571068    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x" event={"ID":"8069f4d6-da2c-4306-8db6-b7fe71262feb","Type":"ContainerDied","Data":"8c063906ed311ff96cb517c3a787d5acd0dc291cd84c2f03b0568a0354806b67"}
Oct 29 23:32:36.570986 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:36.560991    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:36.615351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:36.599466    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:36.960903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:36.960029    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:36.988907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:36.988162    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:37.012903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.012426    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:32:37.120905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.118686    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-openshift-ca\") pod \"8069f4d6-da2c-4306-8db6-b7fe71262feb\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") "
Oct 29 23:32:37.120905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.118777    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8069f4d6-da2c-4306-8db6-b7fe71262feb-kube-api-access-rhvwc\" (UniqueName: \"kubernetes.io/projected/8069f4d6-da2c-4306-8db6-b7fe71262feb-kube-api-access-rhvwc\") pod \"8069f4d6-da2c-4306-8db6-b7fe71262feb\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") "
Oct 29 23:32:37.120905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.118827    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-xks-ca\") pod \"8069f4d6-da2c-4306-8db6-b7fe71262feb\" (UID: \"8069f4d6-da2c-4306-8db6-b7fe71262feb\") "
Oct 29 23:32:37.144920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.141302    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-openshift-ca" pod "8069f4d6-da2c-4306-8db6-b7fe71262feb" (UID: "8069f4d6-da2c-4306-8db6-b7fe71262feb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:37.200916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.180998    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-xks-ca" pod "8069f4d6-da2c-4306-8db6-b7fe71262feb" (UID: "8069f4d6-da2c-4306-8db6-b7fe71262feb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:37.217510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.215987    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:37.217510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.217229    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:37.224697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.224637    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:37.224697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.224672    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8069f4d6-da2c-4306-8db6-b7fe71262feb-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:37.259166 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.259098    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8069f4d6-da2c-4306-8db6-b7fe71262feb-kube-api-access-rhvwc" pod "8069f4d6-da2c-4306-8db6-b7fe71262feb" (UID: "8069f4d6-da2c-4306-8db6-b7fe71262feb"). InnerVolumeSpecName "kube-api-access-rhvwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:37.331923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.325834    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvwc\" (UniqueName: \"kubernetes.io/projected/8069f4d6-da2c-4306-8db6-b7fe71262feb-kube-api-access-rhvwc\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:37.446955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.440635    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"]
Oct 29 23:32:37.613962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.603369    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:37.613962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.605988    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded"
Oct 29 23:32:37.613962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.607252    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 29 23:32:37.613962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.607732    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:37.618381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.615496    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:37.618381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.617464    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:32:37.815925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.814487    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:37.914631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:37.914529    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:32:38.079903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.064032    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Oct 29 23:32:38.174962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.170237    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:32:38.233385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.230900    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:38.233385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.231442    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 29 23:32:38.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.674991    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 23:32:38.707904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.683478    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded"
Oct 29 23:32:38.790858 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.738780    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:38.791510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.738819    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:38.886774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:38.886743    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"
Oct 29 23:32:39.127964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.120095    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:39.171903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.171555    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:32:39.240011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.236783    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:39.263480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.239982    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:39.339522 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:32:39.245166    6203 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.087s"
Oct 29 23:32:39.659440 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.657959    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x" event={"ID":"8069f4d6-da2c-4306-8db6-b7fe71262feb","Type":"ContainerDied","Data":"0c5d551d7d82e6213bd051a9750cc77f211ab4ac249d46e714be60f5d352985b"}
Oct 29 23:32:39.679248 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.679201    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5d551d7d82e6213bd051a9750cc77f211ab4ac249d46e714be60f5d352985b"
Oct 29 23:32:39.679905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.679863    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:39.680337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.680322    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:39.680521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.680498    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:32:39.680674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.680660    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:32:39.691907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.680826    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:32:39.721421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.694484    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 23:32:39.722629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.722603    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:32:39.726901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.723075    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:32:39.747273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.747239    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:32:39.778438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.778411    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:32:39.778794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.778772    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"]
Oct 29 23:32:39.813215 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.813180    6203 cpu_manager.go:414] "containerMap: removing container" podUID="8069f4d6-da2c-4306-8db6-b7fe71262feb" containerName="revoke-keys"
Oct 29 23:32:39.813482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.813458    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8069f4d6-da2c-4306-8db6-b7fe71262feb" containerName="revoke-keys"
Oct 29 23:32:39.813634 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.813620    6203 cpu_manager.go:414] "containerMap: removing container" podUID="19fb5de7-9a1e-4f50-8abd-33a64a830065" containerName="revoke-keys"
Oct 29 23:32:39.813755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.813741    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fb5de7-9a1e-4f50-8abd-33a64a830065" containerName="revoke-keys"
Oct 29 23:32:39.814229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.814212    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="8069f4d6-da2c-4306-8db6-b7fe71262feb" containerName="revoke-keys"
Oct 29 23:32:39.814369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.814356    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fb5de7-9a1e-4f50-8abd-33a64a830065" containerName="revoke-keys"
Oct 29 23:32:39.845684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.845641    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded"
Oct 29 23:32:39.864323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.853962    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 23:32:39.864846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.864807    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6" podUID="415d9077-00ae-4e22-8eed-8e1e0f94e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:40.000917 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:39.997726    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:32:40.123905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.093050    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.149906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.147378    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:40.181952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.178184    6203 patch_prober.go:29] interesting pod/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:32:40.181952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.178256    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:40.295963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.273015    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:32:40.369988 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.364825    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ae1784\"/\"distinct-w6-ae1784-kserve-self-signed-certs\""
Oct 29 23:32:40.458905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.444254    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-model-cache\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.458905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.444416    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-dshm\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.458905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.444475    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpll5\" (UniqueName: \"kubernetes.io/projected/a93027eb-7194-4f70-b09c-d7a767d1715a-kube-api-access-vpll5\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.458905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.444512    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-home\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.458905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.444581    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93027eb-7194-4f70-b09c-d7a767d1715a-tls-certs\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.458905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.444668    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-tmp-dir\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.534908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.531520    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:40.597461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.577945    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93027eb-7194-4f70-b09c-d7a767d1715a-tls-certs\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.597461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.578079    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-tmp-dir\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.597461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.578207    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-model-cache\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.597461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.578347    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-dshm\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.597461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.578402    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpll5\" (UniqueName: \"kubernetes.io/projected/a93027eb-7194-4f70-b09c-d7a767d1715a-kube-api-access-vpll5\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.597461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.578436    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-home\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.597461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.579116    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-home\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:40.642904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.618375    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:32:41.063362 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.672336    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-model-cache\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:41.065950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.672670    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-tmp-dir\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:41.065950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.728959    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 29 23:32:41.066306 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.729423    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:41.066463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.733404    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Oct 29 23:32:41.068244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.836952    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:41.068980 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.838490    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpll5\" (UniqueName: \"kubernetes.io/projected/a93027eb-7194-4f70-b09c-d7a767d1715a-kube-api-access-vpll5\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:41.068980 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.886284    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:41.070026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:40.886316    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:41.070026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.022758    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:41.070403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.024408    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-dshm\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:41.070403 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:32:41.024644    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:32:41.070403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.060412    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"]
Oct 29 23:32:41.070403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.070368    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:41.206515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.180235    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:32:41.206515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.196550    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93027eb-7194-4f70-b09c-d7a767d1715a-tls-certs\") pod \"distinct-w6-ae1784-kserve-6c9f4fb85-xr85w\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") " pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:41.206515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.196649    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:41.245808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.244017    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:32:41.288890 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.288843    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:41.289591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.289339    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:32:41.298058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.298032    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:41.324745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.322038    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:32:41.346558 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.346529    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:32:41.352592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.352564    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:32:41.363058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.363013    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:41.675720 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:41.675689    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda93027eb_7194_4f70_b09c_d7a767d1715a.slice/crio-f356d380fba76ee350646d4a3a052d971b05e2779c2170d8e723d622b4dc7ff8 WatchSource:0}: Error finding container f356d380fba76ee350646d4a3a052d971b05e2779c2170d8e723d622b4dc7ff8: Status 404 returned error can't find the container with id f356d380fba76ee350646d4a3a052d971b05e2779c2170d8e723d622b4dc7ff8
Oct 29 23:32:41.682470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:41.682444    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"]
Oct 29 23:32:42.124211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:42.124035    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" event={"ID":"a93027eb-7194-4f70-b09c-d7a767d1715a","Type":"ContainerStarted","Data":"f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3"}
Oct 29 23:32:42.124211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:42.124096    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" event={"ID":"a93027eb-7194-4f70-b09c-d7a767d1715a","Type":"ContainerStarted","Data":"f356d380fba76ee350646d4a3a052d971b05e2779c2170d8e723d622b4dc7ff8"}
Oct 29 23:32:42.172819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:42.172764    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podStartSLOduration=4.172751439 podStartE2EDuration="4.172751439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:42.171121552 +0000 UTC m=+1140.275381919" watchObservedRunningTime="2026-10-29 23:32:42.172751439 +0000 UTC m=+1140.277011824"
Oct 29 23:32:42.363963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:42.363861    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:43.106555 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:43.106513    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"]
Oct 29 23:32:43.144969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:43.144920    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"]
Oct 29 23:32:45.360383 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.360348    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"]
Oct 29 23:32:45.364859 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.364837    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.390925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.390898    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"]
Oct 29 23:32:45.399640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.399611    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:45.399640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.399616    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:32:45.399863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.399658    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:32:45.399863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.399781    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:32:45.485744 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.485706    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wpbq\" (UniqueName: \"kubernetes.io/projected/8c922007-198e-4db9-bb6a-013860a92683-kube-api-access-9wpbq\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.486011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.485994    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c922007-198e-4db9-bb6a-013860a92683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.486067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.486027    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c922007-198e-4db9-bb6a-013860a92683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.587378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.587342    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c922007-198e-4db9-bb6a-013860a92683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.587518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.587382    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c922007-198e-4db9-bb6a-013860a92683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.587518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.587438    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wpbq\" (UniqueName: \"kubernetes.io/projected/8c922007-198e-4db9-bb6a-013860a92683-kube-api-access-9wpbq\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.588134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.588108    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c922007-198e-4db9-bb6a-013860a92683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.589671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.589648    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c922007-198e-4db9-bb6a-013860a92683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.613923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.613825    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wpbq\" (UniqueName: \"kubernetes.io/projected/8c922007-198e-4db9-bb6a-013860a92683-kube-api-access-9wpbq\") pod \"maas-api-revoke-sub-47fc6e21-xg97k\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.678701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.678668    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:45.755107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.754409    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"]
Oct 29 23:32:45.763221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.762799    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:45.789910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.787999    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"]
Oct 29 23:32:45.897502 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.897409    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:45.897651 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.897547    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qzn\" (UniqueName: \"kubernetes.io/projected/e109ba55-e7cc-48e3-978b-0125b7e7580d-kube-api-access-x8qzn\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:45.897767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.897750    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:45.989287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.989257    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"]
Oct 29 23:32:45.998997 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.998945    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:45.999099 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.999068    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:45.999174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.999144    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qzn\" (UniqueName: \"kubernetes.io/projected/e109ba55-e7cc-48e3-978b-0125b7e7580d-kube-api-access-x8qzn\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:45.999871 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:45.999840    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:46.001408 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:46.001385    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:46.034488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:46.034462    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qzn\" (UniqueName: \"kubernetes.io/projected/e109ba55-e7cc-48e3-978b-0125b7e7580d-kube-api-access-x8qzn\") pod \"maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:46.086285 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:46.086251    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:46.161331 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:46.161290    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k" event={"ID":"8c922007-198e-4db9-bb6a-013860a92683","Type":"ContainerStarted","Data":"8f96709ffdda98809b1cefc27790f1b310efd6ffa040961ebdf3260b0bf4ae69"}
Oct 29 23:32:46.408525 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:32:46.408439    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode109ba55_e7cc_48e3_978b_0125b7e7580d.slice/crio-4bfd6d30d09cc3e9233eecc9b040b1e05ea1b0ecb002e875226cfce5d302bea0 WatchSource:0}: Error finding container 4bfd6d30d09cc3e9233eecc9b040b1e05ea1b0ecb002e875226cfce5d302bea0: Status 404 returned error can't find the container with id 4bfd6d30d09cc3e9233eecc9b040b1e05ea1b0ecb002e875226cfce5d302bea0
Oct 29 23:32:46.410815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:46.410795    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"]
Oct 29 23:32:47.171268 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:47.171220    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8" event={"ID":"e109ba55-e7cc-48e3-978b-0125b7e7580d","Type":"ContainerStarted","Data":"45dbc5be58d003d1b3060492cc322435ac8ac4729a3c1642a6dc28261cccb8ff"}
Oct 29 23:32:47.171486 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:47.171287    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8" event={"ID":"e109ba55-e7cc-48e3-978b-0125b7e7580d","Type":"ContainerStarted","Data":"4bfd6d30d09cc3e9233eecc9b040b1e05ea1b0ecb002e875226cfce5d302bea0"}
Oct 29 23:32:47.173209 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:47.173183    6203 generic.go:356] "Generic (PLEG): container finished" podID="8c922007-198e-4db9-bb6a-013860a92683" containerID="c47d82c475a84afe5a2377506d6ab7d746a9479ac292e2f79c15b8c9d94c5bc7" exitCode=0
Oct 29 23:32:47.173330 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:47.173238    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k" event={"ID":"8c922007-198e-4db9-bb6a-013860a92683","Type":"ContainerDied","Data":"c47d82c475a84afe5a2377506d6ab7d746a9479ac292e2f79c15b8c9d94c5bc7"}
Oct 29 23:32:47.221579 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:47.221532    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8" podStartSLOduration=2.2215189300000002 podStartE2EDuration="2.22151893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:47.219368852 +0000 UTC m=+1145.323629242" watchObservedRunningTime="2026-10-29 23:32:47.22151893 +0000 UTC m=+1145.325779356"
Oct 29 23:32:48.106970 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.106927    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" podUID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" containerName="payload-processing" containerID="cri-o://6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a" gracePeriod=25
Oct 29 23:32:48.145926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.145838    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" podUID="0898b118-c645-49fc-b269-7884fb71d335" containerName="payload-pre-processing" containerID="cri-o://f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1" gracePeriod=25
Oct 29 23:32:48.182410 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.182371    6203 generic.go:356] "Generic (PLEG): container finished" podID="e109ba55-e7cc-48e3-978b-0125b7e7580d" containerID="45dbc5be58d003d1b3060492cc322435ac8ac4729a3c1642a6dc28261cccb8ff" exitCode=0
Oct 29 23:32:48.182595 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.182459    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8" event={"ID":"e109ba55-e7cc-48e3-978b-0125b7e7580d","Type":"ContainerDied","Data":"45dbc5be58d003d1b3060492cc322435ac8ac4729a3c1642a6dc28261cccb8ff"}
Oct 29 23:32:48.353725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.353695    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:48.436179 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.436142    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c922007-198e-4db9-bb6a-013860a92683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c922007-198e-4db9-bb6a-013860a92683-maas-api-openshift-ca\") pod \"8c922007-198e-4db9-bb6a-013860a92683\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") "
Oct 29 23:32:48.436383 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.436329    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c922007-198e-4db9-bb6a-013860a92683-kube-api-access-9wpbq\" (UniqueName: \"kubernetes.io/projected/8c922007-198e-4db9-bb6a-013860a92683-kube-api-access-9wpbq\") pod \"8c922007-198e-4db9-bb6a-013860a92683\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") "
Oct 29 23:32:48.436462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.436383    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c922007-198e-4db9-bb6a-013860a92683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c922007-198e-4db9-bb6a-013860a92683-maas-api-xks-ca\") pod \"8c922007-198e-4db9-bb6a-013860a92683\" (UID: \"8c922007-198e-4db9-bb6a-013860a92683\") "
Oct 29 23:32:48.436554 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.436527    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c922007-198e-4db9-bb6a-013860a92683-maas-api-openshift-ca" pod "8c922007-198e-4db9-bb6a-013860a92683" (UID: "8c922007-198e-4db9-bb6a-013860a92683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:48.437113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.437085    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c922007-198e-4db9-bb6a-013860a92683-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:48.439161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.439127    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c922007-198e-4db9-bb6a-013860a92683-maas-api-xks-ca" pod "8c922007-198e-4db9-bb6a-013860a92683" (UID: "8c922007-198e-4db9-bb6a-013860a92683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:48.439323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.439293    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c922007-198e-4db9-bb6a-013860a92683-kube-api-access-9wpbq" pod "8c922007-198e-4db9-bb6a-013860a92683" (UID: "8c922007-198e-4db9-bb6a-013860a92683"). InnerVolumeSpecName "kube-api-access-9wpbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:48.538835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.538798    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wpbq\" (UniqueName: \"kubernetes.io/projected/8c922007-198e-4db9-bb6a-013860a92683-kube-api-access-9wpbq\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:48.538835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:48.538831    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c922007-198e-4db9-bb6a-013860a92683-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:49.191731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.191699    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"
Oct 29 23:32:49.191731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.191723    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k" event={"ID":"8c922007-198e-4db9-bb6a-013860a92683","Type":"ContainerDied","Data":"8f96709ffdda98809b1cefc27790f1b310efd6ffa040961ebdf3260b0bf4ae69"}
Oct 29 23:32:49.192347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.191781    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f96709ffdda98809b1cefc27790f1b310efd6ffa040961ebdf3260b0bf4ae69"
Oct 29 23:32:49.348134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.348108    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:49.451222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.451140    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-openshift-ca\") pod \"e109ba55-e7cc-48e3-978b-0125b7e7580d\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") "
Oct 29 23:32:49.451392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.451248    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-xks-ca\") pod \"e109ba55-e7cc-48e3-978b-0125b7e7580d\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") "
Oct 29 23:32:49.451392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.451359    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e109ba55-e7cc-48e3-978b-0125b7e7580d-kube-api-access-x8qzn\" (UniqueName: \"kubernetes.io/projected/e109ba55-e7cc-48e3-978b-0125b7e7580d-kube-api-access-x8qzn\") pod \"e109ba55-e7cc-48e3-978b-0125b7e7580d\" (UID: \"e109ba55-e7cc-48e3-978b-0125b7e7580d\") "
Oct 29 23:32:49.451566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.451541    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-openshift-ca" pod "e109ba55-e7cc-48e3-978b-0125b7e7580d" (UID: "e109ba55-e7cc-48e3-978b-0125b7e7580d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:49.451925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.451904    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:49.453369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.453333    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-xks-ca" pod "e109ba55-e7cc-48e3-978b-0125b7e7580d" (UID: "e109ba55-e7cc-48e3-978b-0125b7e7580d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:49.453792 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.453769    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e109ba55-e7cc-48e3-978b-0125b7e7580d-kube-api-access-x8qzn" pod "e109ba55-e7cc-48e3-978b-0125b7e7580d" (UID: "e109ba55-e7cc-48e3-978b-0125b7e7580d"). InnerVolumeSpecName "kube-api-access-x8qzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:49.554506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.554468    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e109ba55-e7cc-48e3-978b-0125b7e7580d-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:49.554506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:49.554504    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8qzn\" (UniqueName: \"kubernetes.io/projected/e109ba55-e7cc-48e3-978b-0125b7e7580d-kube-api-access-x8qzn\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:50.208479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:50.208441    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"
Oct 29 23:32:50.208479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:50.208466    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8" event={"ID":"e109ba55-e7cc-48e3-978b-0125b7e7580d","Type":"ContainerDied","Data":"4bfd6d30d09cc3e9233eecc9b040b1e05ea1b0ecb002e875226cfce5d302bea0"}
Oct 29 23:32:50.209096 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:50.208542    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bfd6d30d09cc3e9233eecc9b040b1e05ea1b0ecb002e875226cfce5d302bea0"
Oct 29 23:32:50.565117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:50.565081    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"]
Oct 29 23:32:51.363740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:51.363700    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:51.384961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:51.384928    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:52.235920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.235894    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:52.918207 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.918172    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"]
Oct 29 23:32:52.919049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.919026    6203 cpu_manager.go:414] "containerMap: removing container" podUID="e109ba55-e7cc-48e3-978b-0125b7e7580d" containerName="revoke-keys"
Oct 29 23:32:52.919049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.919049    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e109ba55-e7cc-48e3-978b-0125b7e7580d" containerName="revoke-keys"
Oct 29 23:32:52.919227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.919088    6203 cpu_manager.go:414] "containerMap: removing container" podUID="8c922007-198e-4db9-bb6a-013860a92683" containerName="revoke-keys"
Oct 29 23:32:52.919227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.919096    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c922007-198e-4db9-bb6a-013860a92683" containerName="revoke-keys"
Oct 29 23:32:52.919333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.919283    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c922007-198e-4db9-bb6a-013860a92683" containerName="revoke-keys"
Oct 29 23:32:52.919333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.919311    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="e109ba55-e7cc-48e3-978b-0125b7e7580d" containerName="revoke-keys"
Oct 29 23:32:52.925538 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.925501    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:52.933936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.933913    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"]
Oct 29 23:32:52.938441 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.938416    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:52.946925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.946899    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"]
Oct 29 23:32:52.957748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.957720    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:32:52.957748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.957757    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:32:52.957748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.957806    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:32:52.958335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.958304    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:32:52.966223 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:52.966197    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"]
Oct 29 23:32:53.110165 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.110121    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.110357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.110212    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.110357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.110253    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gbl\" (UniqueName: \"kubernetes.io/projected/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-kube-api-access-n7gbl\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.110493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.110447    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.110493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.110487    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.110608 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.110562    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqqxz\" (UniqueName: \"kubernetes.io/projected/6a0616a5-b652-4341-9c6e-45c90c51b10f-kube-api-access-jqqxz\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.214692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.212701    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.214692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.212800    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.214692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.212841    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gbl\" (UniqueName: \"kubernetes.io/projected/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-kube-api-access-n7gbl\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.214692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.213048    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.214692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.213101    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.214692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.213190    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqqxz\" (UniqueName: \"kubernetes.io/projected/6a0616a5-b652-4341-9c6e-45c90c51b10f-kube-api-access-jqqxz\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.214692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.214639    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.220515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.217844    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.220515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.218561    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.228251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.228221    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.240279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.240246    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqqxz\" (UniqueName: \"kubernetes.io/projected/6a0616a5-b652-4341-9c6e-45c90c51b10f-kube-api-access-jqqxz\") pod \"maas-api-revoke-sub-24cf3321-8xhbr\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.246912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.246866    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gbl\" (UniqueName: \"kubernetes.io/projected/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-kube-api-access-n7gbl\") pod \"maas-api-revoke-sub-29e8d635-58hpd\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.260250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.260225    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:53.544548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.544446    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:53.611226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:53.592287    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"]
Oct 29 23:32:54.238599 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:54.216909    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"]
Oct 29 23:32:54.298503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:54.298417    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd" event={"ID":"e4f9bf78-eed7-4beb-a4eb-61d51a104f41","Type":"ContainerStarted","Data":"4dfe0365ca1ec4df2fa2e44125c07fe9ab0ba1f3431acb2393f9a06a8ed4ca93"}
Oct 29 23:32:54.301794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:54.301760    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr" event={"ID":"6a0616a5-b652-4341-9c6e-45c90c51b10f","Type":"ContainerStarted","Data":"300f92d522aaf5200144ae38b2e20a5478655b9d2d64073be8b41d35b69bc570"}
Oct 29 23:32:54.402308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:54.401964    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" podUID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:54.404907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:54.404858    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" podUID="0898b118-c645-49fc-b269-7884fb71d335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:32:55.218087 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.200323    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"]
Oct 29 23:32:55.246115 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.231943    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.289820    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.289973    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.290044    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944f6d7f-ecca-4b96-934b-ce03675d1770-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.290084    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.290206    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.290241    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.290337    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vxb8\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-kube-api-access-6vxb8\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.290405    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.291596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.290464    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.380132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.380080    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd" event={"ID":"e4f9bf78-eed7-4beb-a4eb-61d51a104f41","Type":"ContainerStarted","Data":"afcce26e34a8937eb366685c8a6b4f2923f4d463d7482a7b6863e823a8ae8df0"}
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411015    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411098    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411166    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411219    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411287    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944f6d7f-ecca-4b96-934b-ce03675d1770-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411328    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411457    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411484    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.411582    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vxb8\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-kube-api-access-6vxb8\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.418370    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr" event={"ID":"6a0616a5-b652-4341-9c6e-45c90c51b10f","Type":"ContainerStarted","Data":"53419509ea6c226823a6fd2adb96cd6019817f07e752483404ae9b87cc8622cb"}
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.418636    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-credential-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.419347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.418929    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-data\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.437975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.420809    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-socket\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.437975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.433698    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944f6d7f-ecca-4b96-934b-ce03675d1770-istiod-ca-cert\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.464305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.464245    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-envoy\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.536307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.535424    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-certs\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.621960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.600901    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"]
Oct 29 23:32:55.691224 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.691142    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-token\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.747913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.740615    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-podinfo\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.756093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.756014    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vxb8\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-kube-api-access-6vxb8\") pod \"e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") " pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:55.797026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.782661    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr" podStartSLOduration=3.782639812 podStartE2EDuration="3.782639812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:55.50667892 +0000 UTC m=+1153.610939334" watchObservedRunningTime="2026-10-29 23:32:55.782639812 +0000 UTC m=+1153.886900199"
Oct 29 23:32:55.847739 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.815992    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:32:55.897419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.893192    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded"
Oct 29 23:32:55.936643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:55.936572    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd" podStartSLOduration=3.9365481879999997 podStartE2EDuration="3.936548188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:32:55.623701432 +0000 UTC m=+1153.727961820" watchObservedRunningTime="2026-10-29 23:32:55.936548188 +0000 UTC m=+1154.040808575"
Oct 29 23:32:56.176110 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.165194    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:32:56.205902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.199094    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:56.213469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.213428    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:56.323033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.215462    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:56.353834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.247470    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:56.405904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.292818    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:56.452544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.452436    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:56.692904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.665215    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 29 23:32:56.717965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.715486    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" containerID="cri-o://c5e1d1d8275836daf060f1c95616f106cbfda0eb8d14934510d3f922e7e011ec" gracePeriod=41
Oct 29 23:32:56.836049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.836008    6203 generic.go:356] "Generic (PLEG): container finished" podID="e4f9bf78-eed7-4beb-a4eb-61d51a104f41" containerID="afcce26e34a8937eb366685c8a6b4f2923f4d463d7482a7b6863e823a8ae8df0" exitCode=0
Oct 29 23:32:56.837045 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.836764    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd" event={"ID":"e4f9bf78-eed7-4beb-a4eb-61d51a104f41","Type":"ContainerDied","Data":"afcce26e34a8937eb366685c8a6b4f2923f4d463d7482a7b6863e823a8ae8df0"}
Oct 29 23:32:56.878208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.877918    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:32:56.895902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.878560    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:32:56.899764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.898285    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded"
Oct 29 23:32:56.899764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.898969    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:56.933047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.923144    6203 generic.go:356] "Generic (PLEG): container finished" podID="6a0616a5-b652-4341-9c6e-45c90c51b10f" containerID="53419509ea6c226823a6fd2adb96cd6019817f07e752483404ae9b87cc8622cb" exitCode=0
Oct 29 23:32:56.933047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:56.923344    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr" event={"ID":"6a0616a5-b652-4341-9c6e-45c90c51b10f","Type":"ContainerDied","Data":"53419509ea6c226823a6fd2adb96cd6019817f07e752483404ae9b87cc8622cb"}
Oct 29 23:32:57.224905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.216552    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:32:57.236919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.232228    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:32:57.320957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.259103    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:57.376153 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.358961    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:57.391223 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.377313    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:32:57.392116 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.392096    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:32:57.393515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.388621    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:57.395472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.395456    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:57.440904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.433204    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:32:57.440904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.433619    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:57.596961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.571798    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:57.597811 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.583424    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:32:57.598314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.598293    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:32:57.673911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.673400    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:57.701905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.692613    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:32:57.822333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.726478    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": read tcp 10.134.0.2:55224->10.134.0.52:8000: read: connection reset by peer"
Oct 29 23:32:57.822333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.726661    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:32:57.931431 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:57.897149    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:58.087926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.084319    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:58.390006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.382807    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:32:58.484903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.484731    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded"
Oct 29 23:32:58.576145 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.576102    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:58.624060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.578445    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 23:32:58.624060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.579375    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:32:58.693202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.693143    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 29 23:32:58.739029 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.738987    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:32:58.981964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.981304    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 29 23:32:59.026159 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:58.986022    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:32:59.132399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.117747    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:32:59.224370 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.219996    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mbwdj" podUID="5c660259-6736-4175-a257-4e65889e0761" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Oct 29 23:32:59.250039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.230852    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:32:59.250039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.230924    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:32:59.250039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.230966    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:32:59.250039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.237803    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:59.250039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.237923    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:59.250039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.237969    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:32:59.268650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.268488    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:32:59.277621 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.277578    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"]
Oct 29 23:32:59.306816 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.305987    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:32:59.429654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.429619    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:32:59.452836 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.452806    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:32:59.453727 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.453704    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:32:59.455663 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.455619    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 29 23:32:59.495414 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.495380    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"]
Oct 29 23:32:59.526144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.515134    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"]
Oct 29 23:32:59.623813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.622701    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:59.630481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.630454    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:59.643532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.643500    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd" event={"ID":"e4f9bf78-eed7-4beb-a4eb-61d51a104f41","Type":"ContainerDied","Data":"4dfe0365ca1ec4df2fa2e44125c07fe9ab0ba1f3431acb2393f9a06a8ed4ca93"}
Oct 29 23:32:59.643788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.643724    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"
Oct 29 23:32:59.643926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.643738    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dfe0365ca1ec4df2fa2e44125c07fe9ab0ba1f3431acb2393f9a06a8ed4ca93"
Oct 29 23:32:59.646908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.646308    6203 generic.go:356] "Generic (PLEG): container finished" podID="24382701-d877-4546-a484-3a88dcb16266" containerID="c5e1d1d8275836daf060f1c95616f106cbfda0eb8d14934510d3f922e7e011ec" exitCode=0
Oct 29 23:32:59.646908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.646482    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" event={"ID":"24382701-d877-4546-a484-3a88dcb16266","Type":"ContainerDied","Data":"c5e1d1d8275836daf060f1c95616f106cbfda0eb8d14934510d3f922e7e011ec"}
Oct 29 23:32:59.649107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.649083    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" event={"ID":"944f6d7f-ecca-4b96-934b-ce03675d1770","Type":"ContainerStarted","Data":"4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd"}
Oct 29 23:32:59.649284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.649269    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" event={"ID":"944f6d7f-ecca-4b96-934b-ce03675d1770","Type":"ContainerStarted","Data":"40f71eb2d929d55b342075e882a1a41e4dbb39614a1fc8bb6f0d2c2a62429e8b"}
Oct 29 23:32:59.651609 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.651589    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"
Oct 29 23:32:59.651945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.651713    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr" event={"ID":"6a0616a5-b652-4341-9c6e-45c90c51b10f","Type":"ContainerDied","Data":"300f92d522aaf5200144ae38b2e20a5478655b9d2d64073be8b41d35b69bc570"}
Oct 29 23:32:59.651945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.651920    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="300f92d522aaf5200144ae38b2e20a5478655b9d2d64073be8b41d35b69bc570"
Oct 29 23:32:59.700325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.700258    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-kube-api-access-n7gbl\" (UniqueName: \"kubernetes.io/projected/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-kube-api-access-n7gbl\") pod \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") "
Oct 29 23:32:59.700325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.700320    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-xks-ca\") pod \"6a0616a5-b652-4341-9c6e-45c90c51b10f\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") "
Oct 29 23:32:59.700583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.700447    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-openshift-ca\") pod \"6a0616a5-b652-4341-9c6e-45c90c51b10f\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") "
Oct 29 23:32:59.700583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.700533    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-xks-ca\") pod \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") "
Oct 29 23:32:59.700692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.700654    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0616a5-b652-4341-9c6e-45c90c51b10f-kube-api-access-jqqxz\" (UniqueName: \"kubernetes.io/projected/6a0616a5-b652-4341-9c6e-45c90c51b10f-kube-api-access-jqqxz\") pod \"6a0616a5-b652-4341-9c6e-45c90c51b10f\" (UID: \"6a0616a5-b652-4341-9c6e-45c90c51b10f\") "
Oct 29 23:32:59.700692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.700686    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-openshift-ca\") pod \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\" (UID: \"e4f9bf78-eed7-4beb-a4eb-61d51a104f41\") "
Oct 29 23:32:59.701076 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.701045    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-openshift-ca" pod "6a0616a5-b652-4341-9c6e-45c90c51b10f" (UID: "6a0616a5-b652-4341-9c6e-45c90c51b10f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:59.701456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.701429    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.701936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.701912    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-openshift-ca" pod "e4f9bf78-eed7-4beb-a4eb-61d51a104f41" (UID: "e4f9bf78-eed7-4beb-a4eb-61d51a104f41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:32:59.703858 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.703833    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-xks-ca" pod "6a0616a5-b652-4341-9c6e-45c90c51b10f" (UID: "6a0616a5-b652-4341-9c6e-45c90c51b10f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:59.704095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.704053    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-kube-api-access-n7gbl" pod "e4f9bf78-eed7-4beb-a4eb-61d51a104f41" (UID: "e4f9bf78-eed7-4beb-a4eb-61d51a104f41"). InnerVolumeSpecName "kube-api-access-n7gbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:59.707574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.707545    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-xks-ca" pod "e4f9bf78-eed7-4beb-a4eb-61d51a104f41" (UID: "e4f9bf78-eed7-4beb-a4eb-61d51a104f41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:59.708523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.708266    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0616a5-b652-4341-9c6e-45c90c51b10f-kube-api-access-jqqxz" pod "6a0616a5-b652-4341-9c6e-45c90c51b10f" (UID: "6a0616a5-b652-4341-9c6e-45c90c51b10f"). InnerVolumeSpecName "kube-api-access-jqqxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:59.747666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.747632    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802105    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24382701-d877-4546-a484-3a88dcb16266-tls-certs\" (UniqueName: \"kubernetes.io/secret/24382701-d877-4546-a484-3a88dcb16266-tls-certs\") pod \"24382701-d877-4546-a484-3a88dcb16266\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") "
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802173    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-model-cache\") pod \"24382701-d877-4546-a484-3a88dcb16266\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") "
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802507    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-model-cache" pod "24382701-d877-4546-a484-3a88dcb16266" (UID: "24382701-d877-4546-a484-3a88dcb16266"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802663    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-home\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-home\") pod \"24382701-d877-4546-a484-3a88dcb16266\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") "
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802738    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-dshm\") pod \"24382701-d877-4546-a484-3a88dcb16266\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") "
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802815    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-tmp-dir\") pod \"24382701-d877-4546-a484-3a88dcb16266\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") "
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802848    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24382701-d877-4546-a484-3a88dcb16266-kube-api-access-dg9jm\" (UniqueName: \"kubernetes.io/projected/24382701-d877-4546-a484-3a88dcb16266-kube-api-access-dg9jm\") pod \"24382701-d877-4546-a484-3a88dcb16266\" (UID: \"24382701-d877-4546-a484-3a88dcb16266\") "
Oct 29 23:32:59.802953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.802936    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-home" pod "24382701-d877-4546-a484-3a88dcb16266" (UID: "24382701-d877-4546-a484-3a88dcb16266"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:59.803471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803133    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-tmp-dir" pod "24382701-d877-4546-a484-3a88dcb16266" (UID: "24382701-d877-4546-a484-3a88dcb16266"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:59.803653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803625    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.803764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803658    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.803764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803673    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqqxz\" (UniqueName: \"kubernetes.io/projected/6a0616a5-b652-4341-9c6e-45c90c51b10f-kube-api-access-jqqxz\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.803764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803690    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.803764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803707    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7gbl\" (UniqueName: \"kubernetes.io/projected/e4f9bf78-eed7-4beb-a4eb-61d51a104f41-kube-api-access-n7gbl\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.803764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803720    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.803764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803732    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0616a5-b652-4341-9c6e-45c90c51b10f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.803764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.803743    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.805287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.805262    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24382701-d877-4546-a484-3a88dcb16266-tls-certs" pod "24382701-d877-4546-a484-3a88dcb16266" (UID: "24382701-d877-4546-a484-3a88dcb16266"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:32:59.805488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.805465    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24382701-d877-4546-a484-3a88dcb16266-kube-api-access-dg9jm" pod "24382701-d877-4546-a484-3a88dcb16266" (UID: "24382701-d877-4546-a484-3a88dcb16266"). InnerVolumeSpecName "kube-api-access-dg9jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:32:59.805654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.805628    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-dshm" pod "24382701-d877-4546-a484-3a88dcb16266" (UID: "24382701-d877-4546-a484-3a88dcb16266"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:32:59.905437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.905393    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24382701-d877-4546-a484-3a88dcb16266-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.905437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.905434    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24382701-d877-4546-a484-3a88dcb16266-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:32:59.905662 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:32:59.905450    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9jm\" (UniqueName: \"kubernetes.io/projected/24382701-d877-4546-a484-3a88dcb16266-kube-api-access-dg9jm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:00.052411 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.052354    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podStartSLOduration=5.052339106 podStartE2EDuration="5.052339106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:00.050624815 +0000 UTC m=+1158.154885198" watchObservedRunningTime="2026-10-29 23:33:00.052339106 +0000 UTC m=+1158.156599492"
Oct 29 23:33:00.183073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.183040    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:33:00.184608 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.184580    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 23:33:00.184752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.184638    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 23:33:00.681306 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.681279    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"
Oct 29 23:33:00.681813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.681325    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr" event={"ID":"24382701-d877-4546-a484-3a88dcb16266","Type":"ContainerDied","Data":"c9ccd456ab6134c9606c33076a9eb45ae2bc272ac1ccf13072b89acc2a549645"}
Oct 29 23:33:00.681938 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.681851    6203 scope.go:122] "RemoveContainer" containerID="c5e1d1d8275836daf060f1c95616f106cbfda0eb8d14934510d3f922e7e011ec"
Oct 29 23:33:00.763222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.763152    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"]
Oct 29 23:33:00.769827 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.769793    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-53c44e/premium-w5-53c44e-kserve-8466bfd89c-dsfgr"]
Oct 29 23:33:00.997434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.997336    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"]
Oct 29 23:33:00.998049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998029    6203 cpu_manager.go:414] "containerMap: removing container" podUID="6a0616a5-b652-4341-9c6e-45c90c51b10f" containerName="revoke-keys"
Oct 29 23:33:00.998049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998047    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0616a5-b652-4341-9c6e-45c90c51b10f" containerName="revoke-keys"
Oct 29 23:33:00.998228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998067    6203 cpu_manager.go:414] "containerMap: removing container" podUID="e4f9bf78-eed7-4beb-a4eb-61d51a104f41" containerName="revoke-keys"
Oct 29 23:33:00.998228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998073    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f9bf78-eed7-4beb-a4eb-61d51a104f41" containerName="revoke-keys"
Oct 29 23:33:00.998228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998080    6203 cpu_manager.go:414] "containerMap: removing container" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main"
Oct 29 23:33:00.998228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998085    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main"
Oct 29 23:33:00.998228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998181    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0616a5-b652-4341-9c6e-45c90c51b10f" containerName="revoke-keys"
Oct 29 23:33:00.998228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998191    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f9bf78-eed7-4beb-a4eb-61d51a104f41" containerName="revoke-keys"
Oct 29 23:33:00.998228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:00.998231    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="24382701-d877-4546-a484-3a88dcb16266" containerName="main"
Oct 29 23:33:01.003328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.003306    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.026512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.026485    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:33:01.026700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.026595    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:33:01.026700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.026611    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:33:01.026700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.026633    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:33:01.067029 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.066906    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"]
Oct 29 23:33:01.126498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.126464    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.126683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.126529    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.126965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.126941    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgj5\" (UniqueName: \"kubernetes.io/projected/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-kube-api-access-pxgj5\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.183537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.183500    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 23:33:01.183712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.183556    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 23:33:01.229371 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.228359    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgj5\" (UniqueName: \"kubernetes.io/projected/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-kube-api-access-pxgj5\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.229371 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.228508    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.229371 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.228569    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.230680 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.230649    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.256378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.256266    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.260230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.260197    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgj5\" (UniqueName: \"kubernetes.io/projected/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-kube-api-access-pxgj5\") pod \"maas-api-revoke-sub-65701252-ffwwh\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.319587 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.319554    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:01.678183 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:01.678147    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83f02cd_7523_4bbf_8ee9_56cd9e5dbf5e.slice/crio-d0f9aaaf8b025304c6673f869cb31fb034988b07384e15b8ac1b7b096304d2d9 WatchSource:0}: Error finding container d0f9aaaf8b025304c6673f869cb31fb034988b07384e15b8ac1b7b096304d2d9: Status 404 returned error can't find the container with id d0f9aaaf8b025304c6673f869cb31fb034988b07384e15b8ac1b7b096304d2d9
Oct 29 23:33:01.679084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.679061    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"]
Oct 29 23:33:01.693340 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:01.693308    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh" event={"ID":"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e","Type":"ContainerStarted","Data":"d0f9aaaf8b025304c6673f869cb31fb034988b07384e15b8ac1b7b096304d2d9"}
Oct 29 23:33:02.020266 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:02.020185    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24382701-d877-4546-a484-3a88dcb16266" path="/var/lib/kubelet/pods/24382701-d877-4546-a484-3a88dcb16266/volumes"
Oct 29 23:33:02.183590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:02.183558    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 23:33:02.183742 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:02.183612    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 23:33:02.703925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:02.703859    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh" event={"ID":"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e","Type":"ContainerStarted","Data":"94d1c6c8cca2336d8bb88827e6e4361513933e0d9256a28449e897745e9c63c3"}
Oct 29 23:33:02.756472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:02.756410    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh" podStartSLOduration=2.756391045 podStartE2EDuration="2.756391045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:02.754297901 +0000 UTC m=+1160.858558323" watchObservedRunningTime="2026-10-29 23:33:02.756391045 +0000 UTC m=+1160.860651436"
Oct 29 23:33:03.183664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:03.183634    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 23:33:03.183831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:03.183681    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 23:33:03.713662 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:03.713628    6203 generic.go:356] "Generic (PLEG): container finished" podID="b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" containerID="94d1c6c8cca2336d8bb88827e6e4361513933e0d9256a28449e897745e9c63c3" exitCode=0
Oct 29 23:33:03.714063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:03.713692    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh" event={"ID":"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e","Type":"ContainerDied","Data":"94d1c6c8cca2336d8bb88827e6e4361513933e0d9256a28449e897745e9c63c3"}
Oct 29 23:33:03.910998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:03.910963    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"]
Oct 29 23:33:03.915476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:03.915457    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:03.962931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:03.961649    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"]
Oct 29 23:33:04.091495 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.091407    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.091633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.091552    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.091682 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.091625    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6s9\" (UniqueName: \"kubernetes.io/projected/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-kube-api-access-gj6s9\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.183001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.182965    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 23:33:04.183192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.183028    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 23:33:04.193683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.193149    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.193683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.193254    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.193683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.193297    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6s9\" (UniqueName: \"kubernetes.io/projected/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-kube-api-access-gj6s9\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.194443 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.194420    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.196928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.196900    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.251544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.251481    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6s9\" (UniqueName: \"kubernetes.io/projected/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-kube-api-access-gj6s9\") pod \"maas-api-revoke-sub-0b27806e-kjznm\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.395852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.395630    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" podUID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:04.398152 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.398104    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" podUID="0898b118-c645-49fc-b269-7884fb71d335" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:04.527272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.527231    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" podUID="a9f22878-956b-4f01-a384-216eee25c2b8" containerName="payload-pre-processing" containerID="cri-o://3174a0d55d2ae1a296258d0ef078003e4db701ae8325aee8abd0cb874c356552" gracePeriod=25
Oct 29 23:33:04.554058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.548202    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:04.652409 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.651942    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" podUID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" containerName="payload-processing" containerID="cri-o://7eff1d0d9574c407b855ec8edd4e8b668205ff569e150fca72b71f7a6dd8615d" gracePeriod=25
Oct 29 23:33:04.892225 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.891304    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"]
Oct 29 23:33:04.892871 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:04.892837    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2886a9_9a5a_4044_aa00_2b65110ccd6f.slice/crio-03c490d97f5b74e763dc23133ddf8e7ca7e2bb6a69ed77137c36ffb1fc855c0c WatchSource:0}: Error finding container 03c490d97f5b74e763dc23133ddf8e7ca7e2bb6a69ed77137c36ffb1fc855c0c: Status 404 returned error can't find the container with id 03c490d97f5b74e763dc23133ddf8e7ca7e2bb6a69ed77137c36ffb1fc855c0c
Oct 29 23:33:04.920871 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:04.920830    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:05.005803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.005771    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-openshift-ca\") pod \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") "
Oct 29 23:33:05.006020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.005939    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-xks-ca\") pod \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") "
Oct 29 23:33:05.006020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.005982    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-kube-api-access-pxgj5\" (UniqueName: \"kubernetes.io/projected/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-kube-api-access-pxgj5\") pod \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\" (UID: \"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e\") "
Oct 29 23:33:05.006728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.006693    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-openshift-ca" pod "b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" (UID: "b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:05.008492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.008465    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-kube-api-access-pxgj5" pod "b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" (UID: "b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e"). InnerVolumeSpecName "kube-api-access-pxgj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:05.008662 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.008642    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-xks-ca" pod "b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" (UID: "b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:05.108357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.108284    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:05.108357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.108324    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:05.108551 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.108377    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxgj5\" (UniqueName: \"kubernetes.io/projected/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e-kube-api-access-pxgj5\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:05.183358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.183326    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 23:33:05.183509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.183379    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 23:33:05.588973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.586717    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" containerID="cri-o://e9ad17668be42c8753425e0ba4f865edf3cd1d766b8b67ea59b83e67e88d188d" gracePeriod=45
Oct 29 23:33:05.765261 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.765216    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm" event={"ID":"8c2886a9-9a5a-4044-aa00-2b65110ccd6f","Type":"ContainerStarted","Data":"5df0451536e15ba5693ac0d049cd88de2b8c7009bc35fb5bdcb07d7bac844797"}
Oct 29 23:33:05.765426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.765293    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm" event={"ID":"8c2886a9-9a5a-4044-aa00-2b65110ccd6f","Type":"ContainerStarted","Data":"03c490d97f5b74e763dc23133ddf8e7ca7e2bb6a69ed77137c36ffb1fc855c0c"}
Oct 29 23:33:05.768253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.768226    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"
Oct 29 23:33:05.769257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.769231    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh" event={"ID":"b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e","Type":"ContainerDied","Data":"d0f9aaaf8b025304c6673f869cb31fb034988b07384e15b8ac1b7b096304d2d9"}
Oct 29 23:33:05.769303 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.769287    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f9aaaf8b025304c6673f869cb31fb034988b07384e15b8ac1b7b096304d2d9"
Oct 29 23:33:05.770974 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.770955    6203 generic.go:356] "Generic (PLEG): container finished" podID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerID="e9ad17668be42c8753425e0ba4f865edf3cd1d766b8b67ea59b83e67e88d188d" exitCode=0
Oct 29 23:33:05.771018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.770986    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" event={"ID":"5b34fc68-99bf-401e-a88f-087ddcdd077c","Type":"ContainerDied","Data":"e9ad17668be42c8753425e0ba4f865edf3cd1d766b8b67ea59b83e67e88d188d"}
Oct 29 23:33:05.828665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.828610    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm" podStartSLOduration=2.82859758 podStartE2EDuration="2.82859758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:05.823392395 +0000 UTC m=+1163.927652782" watchObservedRunningTime="2026-10-29 23:33:05.82859758 +0000 UTC m=+1163.932857967"
Oct 29 23:33:05.888263 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.888237    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"]
Oct 29 23:33:05.888989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.888971    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" containerName="revoke-keys"
Oct 29 23:33:05.889058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.888990    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" containerName="revoke-keys"
Oct 29 23:33:05.889117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.889108    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" containerName="revoke-keys"
Oct 29 23:33:05.893347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.893326    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:05.912446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.912403    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 23:33:05.912446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.912441    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 23:33:05.912627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.912486    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bbcdn\""
Oct 29 23:33:05.918340 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.918256    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:33:05.925276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:05.925250    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"]
Oct 29 23:33:06.011688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.011406    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 23:33:06.023451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.023414    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-dshm\") pod \"5b34fc68-99bf-401e-a88f-087ddcdd077c\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") "
Oct 29 23:33:06.023619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.023460    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-model-cache\") pod \"5b34fc68-99bf-401e-a88f-087ddcdd077c\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") "
Oct 29 23:33:06.023619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.023505    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-tmp-dir\") pod \"5b34fc68-99bf-401e-a88f-087ddcdd077c\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") "
Oct 29 23:33:06.023757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.023658    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b34fc68-99bf-401e-a88f-087ddcdd077c-kube-api-access-v584j\" (UniqueName: \"kubernetes.io/projected/5b34fc68-99bf-401e-a88f-087ddcdd077c-kube-api-access-v584j\") pod \"5b34fc68-99bf-401e-a88f-087ddcdd077c\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") "
Oct 29 23:33:06.023815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.023762    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b34fc68-99bf-401e-a88f-087ddcdd077c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34fc68-99bf-401e-a88f-087ddcdd077c-tls-certs\") pod \"5b34fc68-99bf-401e-a88f-087ddcdd077c\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") "
Oct 29 23:33:06.023815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.023806    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-home\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-home\") pod \"5b34fc68-99bf-401e-a88f-087ddcdd077c\" (UID: \"5b34fc68-99bf-401e-a88f-087ddcdd077c\") "
Oct 29 23:33:06.024183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024153    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.024277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024220    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-tmp-dir" pod "5b34fc68-99bf-401e-a88f-087ddcdd077c" (UID: "5b34fc68-99bf-401e-a88f-087ddcdd077c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.024277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024238    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9msv\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-kube-api-access-n9msv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.024491 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024465    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.024554 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024513    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.024606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024591    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/def45b17-34b8-4375-aae1-079d041c2b07-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.024756 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024713    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.024821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024771    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.024945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.024925    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.027201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.027126    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b34fc68-99bf-401e-a88f-087ddcdd077c-kube-api-access-v584j" pod "5b34fc68-99bf-401e-a88f-087ddcdd077c" (UID: "5b34fc68-99bf-401e-a88f-087ddcdd077c"). InnerVolumeSpecName "kube-api-access-v584j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:06.027308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.027220    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-home" pod "5b34fc68-99bf-401e-a88f-087ddcdd077c" (UID: "5b34fc68-99bf-401e-a88f-087ddcdd077c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.027308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.027279    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-model-cache" pod "5b34fc68-99bf-401e-a88f-087ddcdd077c" (UID: "5b34fc68-99bf-401e-a88f-087ddcdd077c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.027841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.027819    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-dshm" pod "5b34fc68-99bf-401e-a88f-087ddcdd077c" (UID: "5b34fc68-99bf-401e-a88f-087ddcdd077c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:06.029468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.029438    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b34fc68-99bf-401e-a88f-087ddcdd077c-tls-certs" pod "5b34fc68-99bf-401e-a88f-087ddcdd077c" (UID: "5b34fc68-99bf-401e-a88f-087ddcdd077c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:06.126717 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.126669    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.126928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.126723    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.126928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.126794    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/def45b17-34b8-4375-aae1-079d041c2b07-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.127076 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.126927    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.127250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127205    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.127402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127312    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.127402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127381    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9msv\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-kube-api-access-n9msv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.127542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127482    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v584j\" (UniqueName: \"kubernetes.io/projected/5b34fc68-99bf-401e-a88f-087ddcdd077c-kube-api-access-v584j\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.127542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127504    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b34fc68-99bf-401e-a88f-087ddcdd077c-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.127542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127518    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.127542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127533    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.127739 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127546    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b34fc68-99bf-401e-a88f-087ddcdd077c-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:06.128011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.127990    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.130622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.130096    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/def45b17-34b8-4375-aae1-079d041c2b07-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.130622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.130298    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.130947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.130917    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.131028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.131003    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.155762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.155672    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.162054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.162018    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9msv\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-kube-api-access-n9msv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5zsnh\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.183067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.183029    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 29 23:33:06.183249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.183093    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 29 23:33:06.230405 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.230374    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:06.563854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.563799    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"]
Oct 29 23:33:06.574542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.572401    6203 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:33:06.798139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.797939    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:06.800602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.800570    6203 generic.go:356] "Generic (PLEG): container finished" podID="8c2886a9-9a5a-4044-aa00-2b65110ccd6f" containerID="5df0451536e15ba5693ac0d049cd88de2b8c7009bc35fb5bdcb07d7bac844797" exitCode=0
Oct 29 23:33:06.800750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.800661    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm" event={"ID":"8c2886a9-9a5a-4044-aa00-2b65110ccd6f","Type":"ContainerDied","Data":"5df0451536e15ba5693ac0d049cd88de2b8c7009bc35fb5bdcb07d7bac844797"}
Oct 29 23:33:06.802783 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.802757    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh" event={"ID":"def45b17-34b8-4375-aae1-079d041c2b07","Type":"ContainerStarted","Data":"612f443257613a4ea7865cc6a0e387410e4cca413da5a980cd3e1d80b65b6b85"}
Oct 29 23:33:06.804325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.804302    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"
Oct 29 23:33:06.804466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.804301    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv" event={"ID":"5b34fc68-99bf-401e-a88f-087ddcdd077c","Type":"ContainerDied","Data":"4782271ff01287e1ba31543748aeaa75bad839fa292109349ae09f7698fdb4e2"}
Oct 29 23:33:06.804466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.804429    6203 scope.go:122] "RemoveContainer" containerID="e9ad17668be42c8753425e0ba4f865edf3cd1d766b8b67ea59b83e67e88d188d"
Oct 29 23:33:06.892403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.892368    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"]
Oct 29 23:33:06.897631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:06.897602    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-92f6b3/premium-w3-92f6b3-kserve-776f8fff4c-gs2sv"]
Oct 29 23:33:07.188027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:07.187995    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:33:07.188542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:07.188522    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:33:07.189289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:07.189270    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:33:08.023448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.023412    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" path="/var/lib/kubelet/pods/5b34fc68-99bf-401e-a88f-087ddcdd077c/volumes"
Oct 29 23:33:08.040528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.040500    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:08.161267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.161233    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-kube-api-access-gj6s9\" (UniqueName: \"kubernetes.io/projected/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-kube-api-access-gj6s9\") pod \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") "
Oct 29 23:33:08.161441 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.161381    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-openshift-ca\") pod \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") "
Oct 29 23:33:08.161441 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.161432    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-xks-ca\") pod \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\" (UID: \"8c2886a9-9a5a-4044-aa00-2b65110ccd6f\") "
Oct 29 23:33:08.161806 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.161779    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-openshift-ca" pod "8c2886a9-9a5a-4044-aa00-2b65110ccd6f" (UID: "8c2886a9-9a5a-4044-aa00-2b65110ccd6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:08.162396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.162374    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:08.163653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.163629    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-kube-api-access-gj6s9" pod "8c2886a9-9a5a-4044-aa00-2b65110ccd6f" (UID: "8c2886a9-9a5a-4044-aa00-2b65110ccd6f"). InnerVolumeSpecName "kube-api-access-gj6s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:08.164100 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.164068    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-xks-ca" pod "8c2886a9-9a5a-4044-aa00-2b65110ccd6f" (UID: "8c2886a9-9a5a-4044-aa00-2b65110ccd6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:08.264245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.264186    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:08.264245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.264219    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj6s9\" (UniqueName: \"kubernetes.io/projected/8c2886a9-9a5a-4044-aa00-2b65110ccd6f-kube-api-access-gj6s9\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:08.468615 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.468585    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:33:08.485644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.485615    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:33:08.574411 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.574361    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0898b118-c645-49fc-b269-7884fb71d335-kube-api-access-nshhg\" (UniqueName: \"kubernetes.io/projected/0898b118-c645-49fc-b269-7884fb71d335-kube-api-access-nshhg\") pod \"0898b118-c645-49fc-b269-7884fb71d335\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") "
Oct 29 23:33:08.574575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.574464    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0898b118-c645-49fc-b269-7884fb71d335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0898b118-c645-49fc-b269-7884fb71d335-plugins-config-volume\") pod \"0898b118-c645-49fc-b269-7884fb71d335\" (UID: \"0898b118-c645-49fc-b269-7884fb71d335\") "
Oct 29 23:33:08.574738 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.574715    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88c1e53-6f17-4c53-8b54-fb9880f29b88-kube-api-access-gmlxf\" (UniqueName: \"kubernetes.io/projected/c88c1e53-6f17-4c53-8b54-fb9880f29b88-kube-api-access-gmlxf\") pod \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") "
Oct 29 23:33:08.574919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.574898    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88c1e53-6f17-4c53-8b54-fb9880f29b88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88c1e53-6f17-4c53-8b54-fb9880f29b88-plugins-config-volume\") pod \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\" (UID: \"c88c1e53-6f17-4c53-8b54-fb9880f29b88\") "
Oct 29 23:33:08.575148 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.575126    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0898b118-c645-49fc-b269-7884fb71d335-plugins-config-volume" pod "0898b118-c645-49fc-b269-7884fb71d335" (UID: "0898b118-c645-49fc-b269-7884fb71d335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:08.575408 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.575381    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88c1e53-6f17-4c53-8b54-fb9880f29b88-plugins-config-volume" pod "c88c1e53-6f17-4c53-8b54-fb9880f29b88" (UID: "c88c1e53-6f17-4c53-8b54-fb9880f29b88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:08.575790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.575766    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c88c1e53-6f17-4c53-8b54-fb9880f29b88-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:08.575866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.575796    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0898b118-c645-49fc-b269-7884fb71d335-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:08.581652 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.581619    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0898b118-c645-49fc-b269-7884fb71d335-kube-api-access-nshhg" pod "0898b118-c645-49fc-b269-7884fb71d335" (UID: "0898b118-c645-49fc-b269-7884fb71d335"). InnerVolumeSpecName "kube-api-access-nshhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:08.583220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.583187    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88c1e53-6f17-4c53-8b54-fb9880f29b88-kube-api-access-gmlxf" pod "c88c1e53-6f17-4c53-8b54-fb9880f29b88" (UID: "c88c1e53-6f17-4c53-8b54-fb9880f29b88"). InnerVolumeSpecName "kube-api-access-gmlxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:08.677544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.677499    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmlxf\" (UniqueName: \"kubernetes.io/projected/c88c1e53-6f17-4c53-8b54-fb9880f29b88-kube-api-access-gmlxf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:08.677544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.677543    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nshhg\" (UniqueName: \"kubernetes.io/projected/0898b118-c645-49fc-b269-7884fb71d335-kube-api-access-nshhg\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:08.829016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.828970    6203 generic.go:356] "Generic (PLEG): container finished" podID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" containerID="6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a" exitCode=0
Oct 29 23:33:08.829180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.829024    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"
Oct 29 23:33:08.829180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.829062    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" event={"ID":"c88c1e53-6f17-4c53-8b54-fb9880f29b88","Type":"ContainerDied","Data":"6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a"}
Oct 29 23:33:08.829180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.829120    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx" event={"ID":"c88c1e53-6f17-4c53-8b54-fb9880f29b88","Type":"ContainerDied","Data":"caefc3727a1529d8bd36caeaf8d7051072f89d22337cc07c11055e6fdb93e587"}
Oct 29 23:33:08.829180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.829149    6203 scope.go:122] "RemoveContainer" containerID="6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a"
Oct 29 23:33:08.831294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.831265    6203 generic.go:356] "Generic (PLEG): container finished" podID="0898b118-c645-49fc-b269-7884fb71d335" containerID="f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1" exitCode=0
Oct 29 23:33:08.831400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.831326    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"
Oct 29 23:33:08.831508 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.831480    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" event={"ID":"0898b118-c645-49fc-b269-7884fb71d335","Type":"ContainerDied","Data":"f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1"}
Oct 29 23:33:08.831630 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.831533    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp" event={"ID":"0898b118-c645-49fc-b269-7884fb71d335","Type":"ContainerDied","Data":"af129ec732e996b1323fbd9399a2e4e5d1bc7d386929f23f7dc00c3014acabc7"}
Oct 29 23:33:08.833292 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.833268    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm" event={"ID":"8c2886a9-9a5a-4044-aa00-2b65110ccd6f","Type":"ContainerDied","Data":"03c490d97f5b74e763dc23133ddf8e7ca7e2bb6a69ed77137c36ffb1fc855c0c"}
Oct 29 23:33:08.833388 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.833317    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c490d97f5b74e763dc23133ddf8e7ca7e2bb6a69ed77137c36ffb1fc855c0c"
Oct 29 23:33:08.833441 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.833417    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"
Oct 29 23:33:08.843593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.843569    6203 scope.go:122] "RemoveContainer" containerID="6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a"
Oct 29 23:33:08.844112 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:08.843931    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a\": container with ID starting with 6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a not found: ID does not exist" containerID="6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a"
Oct 29 23:33:08.844112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.843965    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a"} err="failed to get container status \"6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a\": rpc error: code = NotFound desc = could not find container \"6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a\": container with ID starting with 6782b4361b438d29df5351cf9a6f03493db54eaf46f8649c877652d813f2921a not found: ID does not exist"
Oct 29 23:33:08.844112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.843991    6203 scope.go:122] "RemoveContainer" containerID="f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1"
Oct 29 23:33:08.855768 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.855739    6203 scope.go:122] "RemoveContainer" containerID="f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1"
Oct 29 23:33:08.856152 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:08.856125    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1\": container with ID starting with f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1 not found: ID does not exist" containerID="f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1"
Oct 29 23:33:08.856248 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.856162    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1"} err="failed to get container status \"f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1\": rpc error: code = NotFound desc = could not find container \"f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1\": container with ID starting with f96e1d0bcec93b4f2f2457f3f4ea52ee0816ccc3046e5c2e047c68fa734c01e1 not found: ID does not exist"
Oct 29 23:33:08.895332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.895299    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"]
Oct 29 23:33:08.900847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.900817    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-53c44e-85d7bdb54d-8z2jx"]
Oct 29 23:33:08.975317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.974678    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"]
Oct 29 23:33:08.982332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:08.982301    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-53c44e-56b797c97f-zqnhp"]
Oct 29 23:33:09.564032 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.563996    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:09.564369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.564063    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.685447    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"]
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686574    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686595    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686626    6203 cpu_manager.go:414] "containerMap: removing container" podUID="0898b118-c645-49fc-b269-7884fb71d335" containerName="payload-pre-processing"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686635    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0898b118-c645-49fc-b269-7884fb71d335" containerName="payload-pre-processing"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686660    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" containerName="payload-processing"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686670    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" containerName="payload-processing"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686687    6203 cpu_manager.go:414] "containerMap: removing container" podUID="8c2886a9-9a5a-4044-aa00-2b65110ccd6f" containerName="revoke-keys"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686696    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2886a9-9a5a-4044-aa00-2b65110ccd6f" containerName="revoke-keys"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686863    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b34fc68-99bf-401e-a88f-087ddcdd077c" containerName="main"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686898    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="0898b118-c645-49fc-b269-7884fb71d335" containerName="payload-pre-processing"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686910    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" containerName="payload-processing"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.686923    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2886a9-9a5a-4044-aa00-2b65110ccd6f" containerName="revoke-keys"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.696051    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.700398    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.700457    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.700482    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.700527    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.700560    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwms\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-kube-api-access-jbwms\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.700650    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.700742    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.701995    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.722047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.702044    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.730851 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.730807    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"]
Oct 29 23:33:09.803399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803362    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803401    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803433    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803469    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwms\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-kube-api-access-jbwms\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803562    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803656    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803789    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803793    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-credential-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803819    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.803843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803826    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-socket\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.804037 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.803913    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.804118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.804101    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-data\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.804201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.804181    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-certs\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.804586 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.804563    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istiod-ca-cert\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.806126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.806103    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-envoy\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.806229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.806211    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-podinfo\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.829572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.829498    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwms\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-kube-api-access-jbwms\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.830076 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.830054    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-token\") pod \"e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") " pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:09.843407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.843379    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh" event={"ID":"def45b17-34b8-4375-aae1-079d041c2b07","Type":"ContainerStarted","Data":"9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887"}
Oct 29 23:33:09.843523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.843456    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:09.897111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:09.897051    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh" podStartSLOduration=1.907030402 podStartE2EDuration="4.897036562s" totalImagesPullingTime="2.99000616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:09.894993563 +0000 UTC m=+1167.999253948" watchObservedRunningTime="2026-10-29 23:33:09.897036562 +0000 UTC m=+1168.001296947"
Oct 29 23:33:09.927239 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:09.927198    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:33:10.020347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.020317    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0898b118-c645-49fc-b269-7884fb71d335" path="/var/lib/kubelet/pods/0898b118-c645-49fc-b269-7884fb71d335/volumes"
Oct 29 23:33:10.020745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.020730    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88c1e53-6f17-4c53-8b54-fb9880f29b88" path="/var/lib/kubelet/pods/c88c1e53-6f17-4c53-8b54-fb9880f29b88/volumes"
Oct 29 23:33:10.032956 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.032933    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:10.365425 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:10.365380    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61cd131_3432_48e3_9b8f_d0a5990b34f4.slice/crio-8d614f0d8cf6cf818f809207fa251eece20633113fc770c40c5f2e89c0e7004b WatchSource:0}: Error finding container 8d614f0d8cf6cf818f809207fa251eece20633113fc770c40c5f2e89c0e7004b: Status 404 returned error can't find the container with id 8d614f0d8cf6cf818f809207fa251eece20633113fc770c40c5f2e89c0e7004b
Oct 29 23:33:10.366847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.366815    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"]
Oct 29 23:33:10.369313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.369274    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:10.369406 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.369361    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:10.369473 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.369403    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:10.856245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.856206    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" event={"ID":"c61cd131-3432-48e3-9b8f-d0a5990b34f4","Type":"ContainerStarted","Data":"beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f"}
Oct 29 23:33:10.856863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.856269    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" event={"ID":"c61cd131-3432-48e3-9b8f-d0a5990b34f4","Type":"ContainerStarted","Data":"8d614f0d8cf6cf818f809207fa251eece20633113fc770c40c5f2e89c0e7004b"}
Oct 29 23:33:10.858279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.858249    6203 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5zsnh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:33:10.858402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.858306    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh" podUID="def45b17-34b8-4375-aae1-079d041c2b07" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:33:10.941458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:10.941371    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podStartSLOduration=1.941350146 podStartE2EDuration="1.941350146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:10.926512412 +0000 UTC m=+1169.030772808" watchObservedRunningTime="2026-10-29 23:33:10.941350146 +0000 UTC m=+1169.045610532"
Oct 29 23:33:11.037405 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:11.036517    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:11.042592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:11.042549    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 23:33:11.042796 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:11.042623    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 23:33:12.047900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.044335    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 23:33:12.047900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.044402    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 23:33:12.260674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.260640    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"]
Oct 29 23:33:12.265168 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.265147    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.282565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.282537    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:33:12.286919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.286862    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"]
Oct 29 23:33:12.296076 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.296040    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:33:12.296249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.296161    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:33:12.296360 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.296341    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:33:12.370207 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.370117    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.370207 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.370171    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkf2m\" (UniqueName: \"kubernetes.io/projected/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-kube-api-access-gkf2m\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.370483 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.370461    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.472318 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.472281    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.472509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.472389    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.472509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.472429    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf2m\" (UniqueName: \"kubernetes.io/projected/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-kube-api-access-gkf2m\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.473175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.473147    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.474813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.474784    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.501952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.501922    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkf2m\" (UniqueName: \"kubernetes.io/projected/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-kube-api-access-gkf2m\") pod \"maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.582707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.582211    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:12.960203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:12.960169    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"]
Oct 29 23:33:12.960421 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:12.960396    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5b5914_9d4e_45c9_bbb8_7e0eed929184.slice/crio-6145450e6999068ed3b64abecd14dec61efeae1757d66c08d42511b506d29ef0 WatchSource:0}: Error finding container 6145450e6999068ed3b64abecd14dec61efeae1757d66c08d42511b506d29ef0: Status 404 returned error can't find the container with id 6145450e6999068ed3b64abecd14dec61efeae1757d66c08d42511b506d29ef0
Oct 29 23:33:13.047902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:13.046346    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 23:33:13.047902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:13.046458    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 23:33:13.862902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:13.861953    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:33:13.888231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:13.888192    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v" event={"ID":"9a5b5914-9d4e-45c9-bbb8-7e0eed929184","Type":"ContainerStarted","Data":"e71a7c4368800da2305e7aad0662b40b33f042af8c58285963ec0d08d3ac6c9c"}
Oct 29 23:33:13.888499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:13.888480    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v" event={"ID":"9a5b5914-9d4e-45c9-bbb8-7e0eed929184","Type":"ContainerStarted","Data":"6145450e6999068ed3b64abecd14dec61efeae1757d66c08d42511b506d29ef0"}
Oct 29 23:33:13.998237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:13.998179    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v" podStartSLOduration=1.99816254 podStartE2EDuration="1.99816254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:13.993186754 +0000 UTC m=+1172.097447141" watchObservedRunningTime="2026-10-29 23:33:13.99816254 +0000 UTC m=+1172.102422926"
Oct 29 23:33:14.053234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:14.043345    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 23:33:14.053234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:14.043419    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 23:33:14.395920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:14.395845    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" podUID="a9f22878-956b-4f01-a384-216eee25c2b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:14.398053 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:14.398018    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" podUID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:14.899139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:14.899099    6203 generic.go:356] "Generic (PLEG): container finished" podID="9a5b5914-9d4e-45c9-bbb8-7e0eed929184" containerID="e71a7c4368800da2305e7aad0662b40b33f042af8c58285963ec0d08d3ac6c9c" exitCode=0
Oct 29 23:33:14.899339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:14.899180    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v" event={"ID":"9a5b5914-9d4e-45c9-bbb8-7e0eed929184","Type":"ContainerDied","Data":"e71a7c4368800da2305e7aad0662b40b33f042af8c58285963ec0d08d3ac6c9c"}
Oct 29 23:33:15.050901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:15.035301    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 23:33:15.050901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:15.035365    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 23:33:15.275999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:15.275961    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"]
Oct 29 23:33:15.285034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:15.284994    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-53c44e-bfx6x"]
Oct 29 23:33:16.068419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.067829    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 23:33:16.068419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.067918    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 23:33:16.074746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.074205    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8069f4d6-da2c-4306-8db6-b7fe71262feb" path="/var/lib/kubelet/pods/8069f4d6-da2c-4306-8db6-b7fe71262feb/volumes"
Oct 29 23:33:16.320105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.319777    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:16.400503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.400468    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-xks-ca\") pod \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") "
Oct 29 23:33:16.401158 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.400734    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-openshift-ca\") pod \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") "
Oct 29 23:33:16.401158 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.400999    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-kube-api-access-gkf2m\" (UniqueName: \"kubernetes.io/projected/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-kube-api-access-gkf2m\") pod \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\" (UID: \"9a5b5914-9d4e-45c9-bbb8-7e0eed929184\") "
Oct 29 23:33:16.401712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.401685    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-openshift-ca" pod "9a5b5914-9d4e-45c9-bbb8-7e0eed929184" (UID: "9a5b5914-9d4e-45c9-bbb8-7e0eed929184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:16.406018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.405990    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-kube-api-access-gkf2m" pod "9a5b5914-9d4e-45c9-bbb8-7e0eed929184" (UID: "9a5b5914-9d4e-45c9-bbb8-7e0eed929184"). InnerVolumeSpecName "kube-api-access-gkf2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:16.406627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.406595    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-xks-ca" pod "9a5b5914-9d4e-45c9-bbb8-7e0eed929184" (UID: "9a5b5914-9d4e-45c9-bbb8-7e0eed929184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:16.503101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.503069    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:16.503101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.503097    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:16.503101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.503106    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkf2m\" (UniqueName: \"kubernetes.io/projected/9a5b5914-9d4e-45c9-bbb8-7e0eed929184-kube-api-access-gkf2m\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:16.926927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.926384    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v" event={"ID":"9a5b5914-9d4e-45c9-bbb8-7e0eed929184","Type":"ContainerDied","Data":"6145450e6999068ed3b64abecd14dec61efeae1757d66c08d42511b506d29ef0"}
Oct 29 23:33:16.926927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.926449    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6145450e6999068ed3b64abecd14dec61efeae1757d66c08d42511b506d29ef0"
Oct 29 23:33:16.926927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:16.926525    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"
Oct 29 23:33:17.034172 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:17.034130    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 23:33:17.034358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:17.034186    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 23:33:17.220287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:17.220168    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"]
Oct 29 23:33:17.248918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:17.248867    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"]
Oct 29 23:33:18.039446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:18.039413    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:18.040641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:18.040613    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:18.043171 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:18.043148    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:33:23.243316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:23.243281    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"]
Oct 29 23:33:23.293553 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:23.293520    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"]
Oct 29 23:33:23.659059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:23.639089    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:23.747472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:23.746699    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:24.358198 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:24.332063    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 29 23:33:24.669561 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:24.584173    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" podUID="a9f22878-956b-4f01-a384-216eee25c2b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:24.669561 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:24.647011    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:24.757890 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:24.757832    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:25.053966 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.041790    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:33:25.125408 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:25.068308    6203 logging.go:55] [core] [Channel #1147 SubChannel #1149]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:50052", ServerName: "10.134.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:50052: connect: connection refused"
Oct 29 23:33:25.125408 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.102780    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 29 23:33:25.357973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.357829    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:25.369211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.369164    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:25.558112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.558066    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:33:25.777905 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:25.777570    6203 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.687s"
Oct 29 23:33:25.777905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.777693    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"]
Oct 29 23:33:25.777905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.777779    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:33:25.777905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.777821    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:33:25.821021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.820965    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:25.853904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.822512    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:33:26.132124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.851342    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy" containerID="cri-o://9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3" gracePeriod=30
Oct 29 23:33:26.301901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:25.852371    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:33:26.328937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.218673    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:26.331381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.231244    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:26.415958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.375093    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:33:26.415958 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:26.234464    6203 logging.go:55] [core] [Channel #1147 SubChannel #1149]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:50052", ServerName: "10.134.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:50052: connect: connection refused"
Oct 29 23:33:26.415958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.273056    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" podUID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:50052\" within 1s: context deadline exceeded"
Oct 29 23:33:26.415958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.298977    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6" podUID="415d9077-00ae-4e22-8eed-8e1e0f94e1b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:26.443493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.232605    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:33:26.477067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.477015    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 29 23:33:26.569037 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.568990    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:33:26.838430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.838385    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:26.857421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:26.857369    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:27.069993 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.069776    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"]
Oct 29 23:33:27.074048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.074024    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:33:27.074319 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.074308    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:33:27.074425 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.074416    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:33:27.074515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.074506    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"]
Oct 29 23:33:27.085333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.085299    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:33:27.085571 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.085558    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:33:27.085686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.085677    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:33:27.085768 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.085759    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"]
Oct 29 23:33:27.085914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.085902    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:33:27.095637 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.087046    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy" containerID="cri-o://bd048738381018deac50f49006c4272272668e3c83a95c5574cdbe0b26b58f25" gracePeriod=30
Oct 29 23:33:27.097132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.084448    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy" containerID="cri-o://7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e" gracePeriod=30
Oct 29 23:33:27.097341 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.097316    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy" containerID="cri-o://63f9acdeebc29ec5539fbb6c1926360856b782ac720a83f6539b812f846e9091" gracePeriod=30
Oct 29 23:33:27.222141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.220866    6203 generic.go:356] "Generic (PLEG): container finished" podID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" containerID="7eff1d0d9574c407b855ec8edd4e8b668205ff569e150fca72b71f7a6dd8615d" exitCode=0
Oct 29 23:33:27.222518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.222478    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" event={"ID":"d9912d64-c0a0-4f89-90db-e4c43f5e14c5","Type":"ContainerDied","Data":"7eff1d0d9574c407b855ec8edd4e8b668205ff569e150fca72b71f7a6dd8615d"}
Oct 29 23:33:27.234431 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.234403    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:33:27.234903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.234863    6203 generic.go:356] "Generic (PLEG): container finished" podID="a9f22878-956b-4f01-a384-216eee25c2b8" containerID="3174a0d55d2ae1a296258d0ef078003e4db701ae8325aee8abd0cb874c356552" exitCode=0
Oct 29 23:33:27.235939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.235914    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" event={"ID":"a9f22878-956b-4f01-a384-216eee25c2b8","Type":"ContainerDied","Data":"3174a0d55d2ae1a296258d0ef078003e4db701ae8325aee8abd0cb874c356552"}
Oct 29 23:33:27.266754 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.266726    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:33:27.413934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.413284    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:33:27.416904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.414563    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:33:27.436184 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.436148    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:33:27.444437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.444408    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:33:27.567069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.567032    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f22878-956b-4f01-a384-216eee25c2b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f22878-956b-4f01-a384-216eee25c2b8-plugins-config-volume\") pod \"a9f22878-956b-4f01-a384-216eee25c2b8\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") "
Oct 29 23:33:27.567288 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.567155    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f22878-956b-4f01-a384-216eee25c2b8-kube-api-access-pf26l\" (UniqueName: \"kubernetes.io/projected/a9f22878-956b-4f01-a384-216eee25c2b8-kube-api-access-pf26l\") pod \"a9f22878-956b-4f01-a384-216eee25c2b8\" (UID: \"a9f22878-956b-4f01-a384-216eee25c2b8\") "
Oct 29 23:33:27.567659 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.567553    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f22878-956b-4f01-a384-216eee25c2b8-plugins-config-volume" pod "a9f22878-956b-4f01-a384-216eee25c2b8" (UID: "a9f22878-956b-4f01-a384-216eee25c2b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:27.570909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.570850    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f22878-956b-4f01-a384-216eee25c2b8-kube-api-access-pf26l" pod "a9f22878-956b-4f01-a384-216eee25c2b8" (UID: "a9f22878-956b-4f01-a384-216eee25c2b8"). InnerVolumeSpecName "kube-api-access-pf26l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:27.611123 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.611063    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:33:27.668507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.668466    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-plugins-config-volume\") pod \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") "
Oct 29 23:33:27.668824 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.668800    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-kube-api-access-7f4q5\" (UniqueName: \"kubernetes.io/projected/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-kube-api-access-7f4q5\") pod \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\" (UID: \"d9912d64-c0a0-4f89-90db-e4c43f5e14c5\") "
Oct 29 23:33:27.668945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.668910    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-plugins-config-volume" pod "d9912d64-c0a0-4f89-90db-e4c43f5e14c5" (UID: "d9912d64-c0a0-4f89-90db-e4c43f5e14c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:27.669489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.669460    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf26l\" (UniqueName: \"kubernetes.io/projected/a9f22878-956b-4f01-a384-216eee25c2b8-kube-api-access-pf26l\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:27.669489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.669487    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:27.669693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.669502    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f22878-956b-4f01-a384-216eee25c2b8-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:27.671014 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.670981    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-kube-api-access-7f4q5" pod "d9912d64-c0a0-4f89-90db-e4c43f5e14c5" (UID: "d9912d64-c0a0-4f89-90db-e4c43f5e14c5"). InnerVolumeSpecName "kube-api-access-7f4q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:27.770813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:27.770757    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4q5\" (UniqueName: \"kubernetes.io/projected/d9912d64-c0a0-4f89-90db-e4c43f5e14c5-kube-api-access-7f4q5\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:28.244244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.244206    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" podUID="e861dc13-1a62-4d01-94c4-e6e3288416de" containerName="payload-processing" containerID="cri-o://f8c174fb0a1ec584c1a6dc060330e42ebec8b9ccb87abee027dbeb05e333e8ea" gracePeriod=25
Oct 29 23:33:28.245406 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.245380    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"
Oct 29 23:33:28.245544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.245389    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25" event={"ID":"d9912d64-c0a0-4f89-90db-e4c43f5e14c5","Type":"ContainerDied","Data":"f9488805fc53e86bea9e901b8be7a9955cc98e6fbcd383ab90e2b297ead551be"}
Oct 29 23:33:28.245544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.245529    6203 scope.go:122] "RemoveContainer" containerID="7eff1d0d9574c407b855ec8edd4e8b668205ff569e150fca72b71f7a6dd8615d"
Oct 29 23:33:28.247348 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.247321    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"
Oct 29 23:33:28.247451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.247419    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q" event={"ID":"a9f22878-956b-4f01-a384-216eee25c2b8","Type":"ContainerDied","Data":"fe9236fdacb489b6b66bd3a4924a6d810fb4d3ef74486e31652ede18af7fea9b"}
Oct 29 23:33:28.257933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.257916    6203 scope.go:122] "RemoveContainer" containerID="3174a0d55d2ae1a296258d0ef078003e4db701ae8325aee8abd0cb874c356552"
Oct 29 23:33:28.300020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.299944    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" podUID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" containerName="payload-pre-processing" containerID="cri-o://30129b99ffea6d63c32d481a9ce4e7d7ba2aa6204cb3ecd319fa9bc31971e2b5" gracePeriod=25
Oct 29 23:33:28.309335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.309312    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"]
Oct 29 23:33:28.313686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.313664    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-92f6b3-7dbf699cf5-zvs5q"]
Oct 29 23:33:28.352950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.352818    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"]
Oct 29 23:33:28.357123 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:28.357103    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-92f6b3-5f996565b4-9lt25"]
Oct 29 23:33:30.023321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:30.023285    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f22878-956b-4f01-a384-216eee25c2b8" path="/var/lib/kubelet/pods/a9f22878-956b-4f01-a384-216eee25c2b8/volumes"
Oct 29 23:33:30.023771 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:30.023756    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" path="/var/lib/kubelet/pods/d9912d64-c0a0-4f89-90db-e4c43f5e14c5/volumes"
Oct 29 23:33:30.187470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:30.187433    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"]
Oct 29 23:33:30.204439 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:30.204410    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-92f6b3-pjqz8"]
Oct 29 23:33:31.859990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.859965    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:33:31.937507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.937415    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-envoy\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.937507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.937455    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-credential-socket\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.937753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.937585    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-kube-api-access-zkgng\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-kube-api-access-zkgng\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.937753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.937625    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/969fab72-22a9-498b-a688-284edd8f591e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969fab72-22a9-498b-a688-284edd8f591e-istio-podinfo\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.937873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.937807    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-data\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.938296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.937971    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-credential-socket" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:31.938296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938280    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-certs\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.938489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938294    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-data" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:31.938489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938342    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969fab72-22a9-498b-a688-284edd8f591e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969fab72-22a9-498b-a688-284edd8f591e-istiod-ca-cert\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.938489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938376    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-socket\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.938489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938414    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-istio-token\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-istio-token\") pod \"969fab72-22a9-498b-a688-284edd8f591e\" (UID: \"969fab72-22a9-498b-a688-284edd8f591e\") "
Oct 29 23:33:31.938708 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938530    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-certs" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:31.938765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938720    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-socket" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:31.938818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.938765    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969fab72-22a9-498b-a688-284edd8f591e-istiod-ca-cert" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:31.939223 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.939201    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:31.939365 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.939351    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:31.939461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.939450    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/969fab72-22a9-498b-a688-284edd8f591e-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:31.939553 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.939541    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:31.939639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.939627    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:31.941402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.941376    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-envoy" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:31.941510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.941439    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-istio-token" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:31.942375 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.942341    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-kube-api-access-zkgng" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "kube-api-access-zkgng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:31.942849 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:31.942812    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/969fab72-22a9-498b-a688-284edd8f591e-istio-podinfo" pod "969fab72-22a9-498b-a688-284edd8f591e" (UID: "969fab72-22a9-498b-a688-284edd8f591e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:32.021433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.021394    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e109ba55-e7cc-48e3-978b-0125b7e7580d" path="/var/lib/kubelet/pods/e109ba55-e7cc-48e3-978b-0125b7e7580d/volumes"
Oct 29 23:33:32.046438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.046396    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.046438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.046440    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/969fab72-22a9-498b-a688-284edd8f591e-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.046671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.046461    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkgng\" (UniqueName: \"kubernetes.io/projected/969fab72-22a9-498b-a688-284edd8f591e-kube-api-access-zkgng\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.046671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.046476    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/969fab72-22a9-498b-a688-284edd8f591e-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.249519 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.249439    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" containerID="cri-o://f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570" gracePeriod=45
Oct 29 23:33:32.274207 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.273623    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" containerID="cri-o://e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4" gracePeriod=45
Oct 29 23:33:32.281867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.281157    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": EOF"
Oct 29 23:33:32.300693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.300653    6203 generic.go:356] "Generic (PLEG): container finished" podID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerID="bd048738381018deac50f49006c4272272668e3c83a95c5574cdbe0b26b58f25" exitCode=0
Oct 29 23:33:32.300857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.300774    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" event={"ID":"ec0d2e4a-833c-40ef-9d12-8318ec337964","Type":"ContainerDied","Data":"bd048738381018deac50f49006c4272272668e3c83a95c5574cdbe0b26b58f25"}
Oct 29 23:33:32.303447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.303416    6203 generic.go:356] "Generic (PLEG): container finished" podID="969fab72-22a9-498b-a688-284edd8f591e" containerID="9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3" exitCode=0
Oct 29 23:33:32.303619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.303599    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" event={"ID":"969fab72-22a9-498b-a688-284edd8f591e","Type":"ContainerDied","Data":"9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3"}
Oct 29 23:33:32.303698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.303652    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj" event={"ID":"969fab72-22a9-498b-a688-284edd8f591e","Type":"ContainerDied","Data":"1ff7ba732491c6eaaa41a3802ee2a9e6bb0698450c0808413d62cd5990d9a6a0"}
Oct 29 23:33:32.303698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.303684    6203 scope.go:122] "RemoveContainer" containerID="9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3"
Oct 29 23:33:32.303907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.303871    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"
Oct 29 23:33:32.307977 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.307948    6203 generic.go:356] "Generic (PLEG): container finished" podID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerID="63f9acdeebc29ec5539fbb6c1926360856b782ac720a83f6539b812f846e9091" exitCode=0
Oct 29 23:33:32.308167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.308046    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" event={"ID":"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280","Type":"ContainerDied","Data":"63f9acdeebc29ec5539fbb6c1926360856b782ac720a83f6539b812f846e9091"}
Oct 29 23:33:32.418858 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.418828    6203 scope.go:122] "RemoveContainer" containerID="b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef"
Oct 29 23:33:32.422114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.422065    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:33:32.437067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.433538    6203 scope.go:122] "RemoveContainer" containerID="9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3"
Oct 29 23:33:32.437067 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:32.434051    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3\": container with ID starting with 9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3 not found: ID does not exist" containerID="9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3"
Oct 29 23:33:32.437067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.434092    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3"} err="failed to get container status \"9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3\": rpc error: code = NotFound desc = could not find container \"9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3\": container with ID starting with 9d4249b4127248edf59156ba8ced0eb71237a725b62d8a6b8aa0444fbc2ce5a3 not found: ID does not exist"
Oct 29 23:33:32.437067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.434118    6203 scope.go:122] "RemoveContainer" containerID="b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef"
Oct 29 23:33:32.437067 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:32.434741    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef\": container with ID starting with b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef not found: ID does not exist" containerID="b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef"
Oct 29 23:33:32.437067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.434766    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef"} err="failed to get container status \"b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef\": rpc error: code = NotFound desc = could not find container \"b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef\": container with ID starting with b8bdf83eedae1517e71a7ef38aba4a0627a909e56d388121a0f5129539dbfeef not found: ID does not exist"
Oct 29 23:33:32.439447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.439347    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:33:32.474320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.468090    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"]
Oct 29 23:33:32.481637 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.481589    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-fnbjj"]
Oct 29 23:33:32.566364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.565956    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-podinfo\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566085    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-data\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566118    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-credential-socket\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566168    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-token\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-token\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.566364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566249    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-envoy\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566303    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-token\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-token\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566399    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-envoy\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566426    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-socket\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566459    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0d2e4a-833c-40ef-9d12-8318ec337964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0d2e4a-833c-40ef-9d12-8318ec337964-istiod-ca-cert\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566533    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-data\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566604    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-kube-api-access-ntwdl\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-kube-api-access-ntwdl\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566648    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istiod-ca-cert\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566693    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-credential-socket\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566767    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-kube-api-access-zmpmt\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-kube-api-access-zmpmt\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.566912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566804    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-certs\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.567477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566931    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-podinfo\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.567477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.566983    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-certs\") pod \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\" (UID: \"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280\") "
Oct 29 23:33:32.567477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.567012    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-socket\") pod \"ec0d2e4a-833c-40ef-9d12-8318ec337964\" (UID: \"ec0d2e4a-833c-40ef-9d12-8318ec337964\") "
Oct 29 23:33:32.573528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.572704    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-socket" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.573528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.573104    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-socket" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.573528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.573304    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-credential-socket" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.573528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.573395    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.573528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.573421    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.573528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.573532    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-data" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.574095 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.573028    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0d2e4a-833c-40ef-9d12-8318ec337964-istiod-ca-cert" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:32.576403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.574310    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-credential-socket" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.576403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.574496    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-data" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.576403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.575005    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-certs" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.576403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.575849    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-certs" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.581220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.581060    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-token" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.581220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.581174    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istiod-ca-cert" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:32.582818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.582773    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-kube-api-access-zmpmt" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "kube-api-access-zmpmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.585114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.585067    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-kube-api-access-ntwdl" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "kube-api-access-ntwdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.585114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.585077    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-envoy" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.585114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.585069    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-token" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.585465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.585438    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-podinfo" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:32.586118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.586090    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-envoy" pod "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" (UID: "ed7aa6a8-95b4-4b68-b1a2-4a91b285a280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.586964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.586924    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-podinfo" pod "ec0d2e4a-833c-40ef-9d12-8318ec337964" (UID: "ec0d2e4a-833c-40ef-9d12-8318ec337964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:32.615692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.615593    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:33:32.652773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.652743    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:33:32.674827 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674787    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.674827 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674828    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0d2e4a-833c-40ef-9d12-8318ec337964-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674842    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674855    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntwdl\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-kube-api-access-ntwdl\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674870    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674901    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674915    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmpmt\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-kube-api-access-zmpmt\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674929    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0d2e4a-833c-40ef-9d12-8318ec337964-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674941    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674953    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674965    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674976    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674988    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.674999    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0d2e4a-833c-40ef-9d12-8318ec337964-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.675014    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.675069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.675025    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.680669 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.680645    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:33:32.775656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775571    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73462746-78dd-4ec9-8d31-5e9fb54d7945-tls-certs\" (UniqueName: \"kubernetes.io/secret/73462746-78dd-4ec9-8d31-5e9fb54d7945-tls-certs\") pod \"73462746-78dd-4ec9-8d31-5e9fb54d7945\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") "
Oct 29 23:33:32.775656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775610    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-dshm\") pod \"73462746-78dd-4ec9-8d31-5e9fb54d7945\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") "
Oct 29 23:33:32.775656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775639    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-data\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.775971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775697    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istiod-ca-cert\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.775971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775750    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-home\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-home\") pod \"73462746-78dd-4ec9-8d31-5e9fb54d7945\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") "
Oct 29 23:33:32.775971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775818    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-podinfo\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.775971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775869    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-home\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-home\") pod \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") "
Oct 29 23:33:32.775971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775925    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tmp-dir\") pod \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") "
Oct 29 23:33:32.775971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775946    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-data" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.775971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.775964    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-envoy\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.776353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776008    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-kube-api-access-nwjz7\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-kube-api-access-nwjz7\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.776353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776035    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tls-certs\") pod \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") "
Oct 29 23:33:32.776353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776111    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-tmp-dir\") pod \"73462746-78dd-4ec9-8d31-5e9fb54d7945\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") "
Oct 29 23:33:32.776353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776173    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-token\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-token\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.776353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776233    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-model-cache\") pod \"73462746-78dd-4ec9-8d31-5e9fb54d7945\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776431    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-model-cache\") pod \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776476    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-kube-api-access-lgrl8\" (UniqueName: \"kubernetes.io/projected/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-kube-api-access-lgrl8\") pod \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776545    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-socket\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776583    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-certs\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776703    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-credential-socket\") pod \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\" (UID: \"ad364fdf-8cdf-419b-b95d-35d20e57da3a\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776745    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73462746-78dd-4ec9-8d31-5e9fb54d7945-kube-api-access-gtv59\" (UniqueName: \"kubernetes.io/projected/73462746-78dd-4ec9-8d31-5e9fb54d7945-kube-api-access-gtv59\") pod \"73462746-78dd-4ec9-8d31-5e9fb54d7945\" (UID: \"73462746-78dd-4ec9-8d31-5e9fb54d7945\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776794    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-dshm\") pod \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\" (UID: \"9b8b6c76-2dfb-4c01-a46d-9b88a001d994\") "
Oct 29 23:33:32.777898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.777591    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.778687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776249    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istiod-ca-cert" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:32.778687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776347    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-home" pod "73462746-78dd-4ec9-8d31-5e9fb54d7945" (UID: "73462746-78dd-4ec9-8d31-5e9fb54d7945"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.778687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.776729    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-tmp-dir" pod "73462746-78dd-4ec9-8d31-5e9fb54d7945" (UID: "73462746-78dd-4ec9-8d31-5e9fb54d7945"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.778687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.777563    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-certs" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.779020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.778929    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73462746-78dd-4ec9-8d31-5e9fb54d7945-tls-certs" pod "73462746-78dd-4ec9-8d31-5e9fb54d7945" (UID: "73462746-78dd-4ec9-8d31-5e9fb54d7945"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:32.779098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.779072    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-socket" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.779281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.779237    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-model-cache" pod "9b8b6c76-2dfb-4c01-a46d-9b88a001d994" (UID: "9b8b6c76-2dfb-4c01-a46d-9b88a001d994"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.779368 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.779322    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-model-cache" pod "73462746-78dd-4ec9-8d31-5e9fb54d7945" (UID: "73462746-78dd-4ec9-8d31-5e9fb54d7945"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.779433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.779370    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tmp-dir" pod "9b8b6c76-2dfb-4c01-a46d-9b88a001d994" (UID: "9b8b6c76-2dfb-4c01-a46d-9b88a001d994"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.779994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.779960    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-kube-api-access-nwjz7" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "kube-api-access-nwjz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.780342 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.780316    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-home" pod "9b8b6c76-2dfb-4c01-a46d-9b88a001d994" (UID: "9b8b6c76-2dfb-4c01-a46d-9b88a001d994"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.780489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.780409    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-credential-socket" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.781345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.781319    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73462746-78dd-4ec9-8d31-5e9fb54d7945-kube-api-access-gtv59" pod "73462746-78dd-4ec9-8d31-5e9fb54d7945" (UID: "73462746-78dd-4ec9-8d31-5e9fb54d7945"). InnerVolumeSpecName "kube-api-access-gtv59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.783403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.783360    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-dshm" pod "9b8b6c76-2dfb-4c01-a46d-9b88a001d994" (UID: "9b8b6c76-2dfb-4c01-a46d-9b88a001d994"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.783403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.783373    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tls-certs" pod "9b8b6c76-2dfb-4c01-a46d-9b88a001d994" (UID: "9b8b6c76-2dfb-4c01-a46d-9b88a001d994"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:32.783802 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.783623    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-envoy" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.784942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.784580    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-token" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.785061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.785038    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-podinfo" pod "ad364fdf-8cdf-419b-b95d-35d20e57da3a" (UID: "ad364fdf-8cdf-419b-b95d-35d20e57da3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:32.785122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.785052    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-dshm" pod "73462746-78dd-4ec9-8d31-5e9fb54d7945" (UID: "73462746-78dd-4ec9-8d31-5e9fb54d7945"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:32.785122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.785089    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-kube-api-access-lgrl8" pod "9b8b6c76-2dfb-4c01-a46d-9b88a001d994" (UID: "9b8b6c76-2dfb-4c01-a46d-9b88a001d994"). InnerVolumeSpecName "kube-api-access-lgrl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:32.879320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879279    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwjz7\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-kube-api-access-nwjz7\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879316    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879332    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879346    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879360    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879373    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879386    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgrl8\" (UniqueName: \"kubernetes.io/projected/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-kube-api-access-lgrl8\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879402    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879416    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879430    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879444    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtv59\" (UniqueName: \"kubernetes.io/projected/73462746-78dd-4ec9-8d31-5e9fb54d7945-kube-api-access-gtv59\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879458    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879471    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73462746-78dd-4ec9-8d31-5e9fb54d7945-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879485    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879498    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879510    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73462746-78dd-4ec9-8d31-5e9fb54d7945-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879523    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879536    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879549    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8b6c76-2dfb-4c01-a46d-9b88a001d994-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:32.879734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:32.879562    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad364fdf-8cdf-419b-b95d-35d20e57da3a-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:33.317962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.317932    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"
Oct 29 23:33:33.318140 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.317982    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772" event={"ID":"ec0d2e4a-833c-40ef-9d12-8318ec337964","Type":"ContainerDied","Data":"0103c319bc7c3eef5ab18445e4fda8b268fcabd856ad2e52781afaba17051931"}
Oct 29 23:33:33.318140 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.318043    6203 scope.go:122] "RemoveContainer" containerID="bd048738381018deac50f49006c4272272668e3c83a95c5574cdbe0b26b58f25"
Oct 29 23:33:33.320967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.320942    6203 generic.go:356] "Generic (PLEG): container finished" podID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerID="f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570" exitCode=0
Oct 29 23:33:33.321107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.320977    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"
Oct 29 23:33:33.321107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.321003    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" event={"ID":"9b8b6c76-2dfb-4c01-a46d-9b88a001d994","Type":"ContainerDied","Data":"f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570"}
Oct 29 23:33:33.321107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.321034    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" event={"ID":"9b8b6c76-2dfb-4c01-a46d-9b88a001d994","Type":"ContainerDied","Data":"19c38d360943ceff3993e44dd98d601951c7b49daa07592bb7b2a6b6c480e511"}
Oct 29 23:33:33.323303 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.323282    6203 generic.go:356] "Generic (PLEG): container finished" podID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerID="7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e" exitCode=0
Oct 29 23:33:33.323401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.323320    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"
Oct 29 23:33:33.323401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.323389    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" event={"ID":"ad364fdf-8cdf-419b-b95d-35d20e57da3a","Type":"ContainerDied","Data":"7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e"}
Oct 29 23:33:33.323588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.323431    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64" event={"ID":"ad364fdf-8cdf-419b-b95d-35d20e57da3a","Type":"ContainerDied","Data":"b586585ad109fd8ec998cd74ea7c6b947fbee8766a6b55a5a14ae3d42a381351"}
Oct 29 23:33:33.325157 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.325132    6203 generic.go:356] "Generic (PLEG): container finished" podID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerID="e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4" exitCode=0
Oct 29 23:33:33.325251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.325171    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"
Oct 29 23:33:33.325251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.325235    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" event={"ID":"73462746-78dd-4ec9-8d31-5e9fb54d7945","Type":"ContainerDied","Data":"e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4"}
Oct 29 23:33:33.325367 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.325263    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg" event={"ID":"73462746-78dd-4ec9-8d31-5e9fb54d7945","Type":"ContainerDied","Data":"49145fad01ed976a82e8d4037d52132e0ed36af903af91f9b2a711082ca3e7af"}
Oct 29 23:33:33.326972 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.326910    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf" event={"ID":"ed7aa6a8-95b4-4b68-b1a2-4a91b285a280","Type":"ContainerDied","Data":"613126627ff12e7796ced059c74a42f5891d3947f4adcb0eb99fdb0c1143b31f"}
Oct 29 23:33:33.326972 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.326936    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"
Oct 29 23:33:33.331506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.331484    6203 scope.go:122] "RemoveContainer" containerID="f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570"
Oct 29 23:33:33.344990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.344968    6203 scope.go:122] "RemoveContainer" containerID="f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570"
Oct 29 23:33:33.345319 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:33.345300    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570\": container with ID starting with f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570 not found: ID does not exist" containerID="f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570"
Oct 29 23:33:33.345392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.345332    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570"} err="failed to get container status \"f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570\": rpc error: code = NotFound desc = could not find container \"f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570\": container with ID starting with f8fb1af6574467fc84604308d796e3b3fcb6a641f5a6e7986575b2a2c1303570 not found: ID does not exist"
Oct 29 23:33:33.345392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.345355    6203 scope.go:122] "RemoveContainer" containerID="7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e"
Oct 29 23:33:33.355175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.355154    6203 scope.go:122] "RemoveContainer" containerID="7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e"
Oct 29 23:33:33.355428 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:33.355412    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e\": container with ID starting with 7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e not found: ID does not exist" containerID="7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e"
Oct 29 23:33:33.355477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.355437    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e"} err="failed to get container status \"7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e\": rpc error: code = NotFound desc = could not find container \"7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e\": container with ID starting with 7fb1257e11cfe345469bbf2a4f49ad72f240be478f04437544dfe7e2360eb06e not found: ID does not exist"
Oct 29 23:33:33.355477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.355452    6203 scope.go:122] "RemoveContainer" containerID="e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4"
Oct 29 23:33:33.364751 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.364723    6203 scope.go:122] "RemoveContainer" containerID="e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4"
Oct 29 23:33:33.365045 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:33.365024    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4\": container with ID starting with e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4 not found: ID does not exist" containerID="e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4"
Oct 29 23:33:33.365116 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.365051    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4"} err="failed to get container status \"e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4\": rpc error: code = NotFound desc = could not find container \"e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4\": container with ID starting with e50956e8df15cb9db8373abaa0a1c84241666d8899260c450bbee8c7cd755ed4 not found: ID does not exist"
Oct 29 23:33:33.365116 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.365066    6203 scope.go:122] "RemoveContainer" containerID="63f9acdeebc29ec5539fbb6c1926360856b782ac720a83f6539b812f846e9091"
Oct 29 23:33:33.399254 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.399212    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"]
Oct 29 23:33:33.404976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.404950    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-sk772"]
Oct 29 23:33:33.445749 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.445718    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"]
Oct 29 23:33:33.449858 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.449834    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-z2v64"]
Oct 29 23:33:33.504925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.504889    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"]
Oct 29 23:33:33.517034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.514924    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m"]
Oct 29 23:33:33.550871 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.550836    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"]
Oct 29 23:33:33.555368 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.555333    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b978a0/simulator-w0-b978a0-kserve-6dcffb4fcd-z9ssg"]
Oct 29 23:33:33.581048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.580967    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b978a0/distinct-w0-b978a0-kserve-64b56ff8fd-9h92m" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 29 23:33:33.593792 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.593759    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"]
Oct 29 23:33:33.600551 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:33.600520    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-53c44e-openshift-default-6954bffb8c-4d9hf"]
Oct 29 23:33:34.020357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.020322    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" path="/var/lib/kubelet/pods/73462746-78dd-4ec9-8d31-5e9fb54d7945/volumes"
Oct 29 23:33:34.020864 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.020803    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969fab72-22a9-498b-a688-284edd8f591e" path="/var/lib/kubelet/pods/969fab72-22a9-498b-a688-284edd8f591e/volumes"
Oct 29 23:33:34.021236 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.021223    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" path="/var/lib/kubelet/pods/9b8b6c76-2dfb-4c01-a46d-9b88a001d994/volumes"
Oct 29 23:33:34.021975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.021960    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" path="/var/lib/kubelet/pods/ad364fdf-8cdf-419b-b95d-35d20e57da3a/volumes"
Oct 29 23:33:34.022393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.022381    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" path="/var/lib/kubelet/pods/ec0d2e4a-833c-40ef-9d12-8318ec337964/volumes"
Oct 29 23:33:34.022782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.022770    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" path="/var/lib/kubelet/pods/ed7aa6a8-95b4-4b68-b1a2-4a91b285a280/volumes"
Oct 29 23:33:34.396210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.396116    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" podUID="e861dc13-1a62-4d01-94c4-e6e3288416de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:34.396805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:34.396778    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" podUID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:35.939932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.939891    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"]
Oct 29 23:33:35.940672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940650    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy"
Oct 29 23:33:35.940714 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940675    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy"
Oct 29 23:33:35.940714 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940702    6203 cpu_manager.go:414] "containerMap: removing container" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main"
Oct 29 23:33:35.940714 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940710    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940730    6203 cpu_manager.go:414] "containerMap: removing container" podUID="9a5b5914-9d4e-45c9-bbb8-7e0eed929184" containerName="revoke-keys"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940739    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5b5914-9d4e-45c9-bbb8-7e0eed929184" containerName="revoke-keys"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940753    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940761    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940773    6203 cpu_manager.go:414] "containerMap: removing container" podUID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" containerName="payload-processing"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940779    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" containerName="payload-processing"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940791    6203 cpu_manager.go:414] "containerMap: removing container" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940796    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940814    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940821    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940837    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a9f22878-956b-4f01-a384-216eee25c2b8" containerName="payload-pre-processing"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940846    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f22878-956b-4f01-a384-216eee25c2b8" containerName="payload-pre-processing"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940858    6203 cpu_manager.go:414] "containerMap: removing container" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy"
Oct 29 23:33:35.940832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940866    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940892    6203 cpu_manager.go:414] "containerMap: removing container" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.940900    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941056    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad364fdf-8cdf-419b-b95d-35d20e57da3a" containerName="istio-proxy"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941072    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8b6c76-2dfb-4c01-a46d-9b88a001d994" containerName="main"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941080    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5b5914-9d4e-45c9-bbb8-7e0eed929184" containerName="revoke-keys"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941086    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7aa6a8-95b4-4b68-b1a2-4a91b285a280" containerName="istio-proxy"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941091    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9912d64-c0a0-4f89-90db-e4c43f5e14c5" containerName="payload-processing"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941098    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="73462746-78dd-4ec9-8d31-5e9fb54d7945" containerName="main"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941109    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0d2e4a-833c-40ef-9d12-8318ec337964" containerName="istio-proxy"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941120    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f22878-956b-4f01-a384-216eee25c2b8" containerName="payload-pre-processing"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941129    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy"
Oct 29 23:33:35.941382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.941135    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="969fab72-22a9-498b-a688-284edd8f591e" containerName="istio-proxy"
Oct 29 23:33:35.946758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.946734    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:35.965304 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.965274    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"]
Oct 29 23:33:35.971423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.971396    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"]
Oct 29 23:33:35.971574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.971490    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:35.974542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.974406    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:33:35.974542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.974461    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:33:35.974741 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.974564    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:33:35.974741 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.974567    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:33:35.995271 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:35.995237    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"]
Oct 29 23:33:36.125575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.125540    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpb24\" (UniqueName: \"kubernetes.io/projected/c0a12187-4302-46eb-b9b5-00b18f1c8171-kube-api-access-hpb24\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.125575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.125579    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.125775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.125599    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99gkp\" (UniqueName: \"kubernetes.io/projected/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-kube-api-access-99gkp\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.126048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.126013    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.126426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.126397    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.126537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.126512    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.227685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.227599    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpb24\" (UniqueName: \"kubernetes.io/projected/c0a12187-4302-46eb-b9b5-00b18f1c8171-kube-api-access-hpb24\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.227685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.227651    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.227685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.227675    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99gkp\" (UniqueName: \"kubernetes.io/projected/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-kube-api-access-99gkp\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.228008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.227844    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.228008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.227923    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.228008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.227999    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.228646 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.228621    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.228646 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.228641    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.230302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.230276    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.230577 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.230556    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.266529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.266502    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99gkp\" (UniqueName: \"kubernetes.io/projected/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-kube-api-access-99gkp\") pod \"maas-api-revoke-sub-daf1a463-cndv4\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.266673 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.266568    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpb24\" (UniqueName: \"kubernetes.io/projected/c0a12187-4302-46eb-b9b5-00b18f1c8171-kube-api-access-hpb24\") pod \"maas-api-revoke-sub-134e1509-ksbph\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.293558 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.293531    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:36.560942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.560825    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:36.620440 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.620404    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"]
Oct 29 23:33:36.620552 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:36.620516    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7340bf_2bf3_4cb0_a9d8_889850892bd2.slice/crio-6d343be304865bb92490a1d5838a3c50691621b07356c94f8cb0cf893924ecce WatchSource:0}: Error finding container 6d343be304865bb92490a1d5838a3c50691621b07356c94f8cb0cf893924ecce: Status 404 returned error can't find the container with id 6d343be304865bb92490a1d5838a3c50691621b07356c94f8cb0cf893924ecce
Oct 29 23:33:36.869223 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:36.869195    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a12187_4302_46eb_b9b5_00b18f1c8171.slice/crio-5010612cd050084f5af0c86e8381c0f1140e912e9a0b472b1bf39a8f6655bcf5 WatchSource:0}: Error finding container 5010612cd050084f5af0c86e8381c0f1140e912e9a0b472b1bf39a8f6655bcf5: Status 404 returned error can't find the container with id 5010612cd050084f5af0c86e8381c0f1140e912e9a0b472b1bf39a8f6655bcf5
Oct 29 23:33:36.870428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:36.870394    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"]
Oct 29 23:33:37.396989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:37.396861    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4" event={"ID":"ae7340bf-2bf3-4cb0-a9d8-889850892bd2","Type":"ContainerStarted","Data":"b2894d6f4141690646ae5d59b516ceb2a7ab389e19e6b7672ab7cd50d275da39"}
Oct 29 23:33:37.396989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:37.396954    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4" event={"ID":"ae7340bf-2bf3-4cb0-a9d8-889850892bd2","Type":"ContainerStarted","Data":"6d343be304865bb92490a1d5838a3c50691621b07356c94f8cb0cf893924ecce"}
Oct 29 23:33:37.398687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:37.398648    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph" event={"ID":"c0a12187-4302-46eb-b9b5-00b18f1c8171","Type":"ContainerStarted","Data":"5afb081f6d07f156917cc6ab7b9def4815fe5ec160288829a0020f72dfaacae2"}
Oct 29 23:33:37.399193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:37.398695    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph" event={"ID":"c0a12187-4302-46eb-b9b5-00b18f1c8171","Type":"ContainerStarted","Data":"5010612cd050084f5af0c86e8381c0f1140e912e9a0b472b1bf39a8f6655bcf5"}
Oct 29 23:33:37.451712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:37.451665    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4" podStartSLOduration=2.451651375 podStartE2EDuration="2.451651375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:37.449982507 +0000 UTC m=+1195.554242893" watchObservedRunningTime="2026-10-29 23:33:37.451651375 +0000 UTC m=+1195.555911792"
Oct 29 23:33:37.503325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:37.503264    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph" podStartSLOduration=2.503248151 podStartE2EDuration="2.503248151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:37.500352754 +0000 UTC m=+1195.604613140" watchObservedRunningTime="2026-10-29 23:33:37.503248151 +0000 UTC m=+1195.607508538"
Oct 29 23:33:38.414101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.414062    6203 generic.go:356] "Generic (PLEG): container finished" podID="c0a12187-4302-46eb-b9b5-00b18f1c8171" containerID="5afb081f6d07f156917cc6ab7b9def4815fe5ec160288829a0020f72dfaacae2" exitCode=0
Oct 29 23:33:38.415552 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.414135    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph" event={"ID":"c0a12187-4302-46eb-b9b5-00b18f1c8171","Type":"ContainerDied","Data":"5afb081f6d07f156917cc6ab7b9def4815fe5ec160288829a0020f72dfaacae2"}
Oct 29 23:33:38.425124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.424927    6203 generic.go:356] "Generic (PLEG): container finished" podID="ae7340bf-2bf3-4cb0-a9d8-889850892bd2" containerID="b2894d6f4141690646ae5d59b516ceb2a7ab389e19e6b7672ab7cd50d275da39" exitCode=0
Oct 29 23:33:38.425124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.425072    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4" event={"ID":"ae7340bf-2bf3-4cb0-a9d8-889850892bd2","Type":"ContainerDied","Data":"b2894d6f4141690646ae5d59b516ceb2a7ab389e19e6b7672ab7cd50d275da39"}
Oct 29 23:33:38.523218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.523159    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"]
Oct 29 23:33:38.524548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.524486    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy" containerID="cri-o://6d130e084323b11dc8df3a5b08d8c38a1e1d50e0a102e786bc1ce9070513ca7c" gracePeriod=30
Oct 29 23:33:38.529522 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.529458    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"]
Oct 29 23:33:38.529787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.529754    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" containerID="cri-o://757122e7823460841fca55431016fde9e927cb96416b57b858f94ecb6b65aa13" gracePeriod=30
Oct 29 23:33:38.535041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.535013    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"]
Oct 29 23:33:38.535263 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:38.535235    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy" containerID="cri-o://19f8bcda38c8df72ad65b841442d0fbaec40b3b3c43a5a893d03e6d31889175b" gracePeriod=30
Oct 29 23:33:39.224708 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.224669    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"]
Oct 29 23:33:39.229164 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.229140    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.262475 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.262435    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"]
Oct 29 23:33:39.293367 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293329    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293397    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcnq9\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-kube-api-access-mcnq9\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293498    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293537    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293567    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293658    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293769 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293756    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293825    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.293915 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.293846    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1138d3a-a1ba-4384-8521-920fc1b1339a-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395333    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395402    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcnq9\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-kube-api-access-mcnq9\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395467    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395503    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395543    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395578    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395693    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395787    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.395918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.395819    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1138d3a-a1ba-4384-8521-920fc1b1339a-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.396080 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.396046    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.396463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.396345    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.396463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.396406    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.396463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.396445    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.396671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.396631    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1138d3a-a1ba-4384-8521-920fc1b1339a-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.398517 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.398488    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.398722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.398705    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.421573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.421539    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.421903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.421751    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcnq9\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-kube-api-access-mcnq9\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.547058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.547023    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:39.608274 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.607852    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:39.633219 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.633194    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:39.700479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.700396    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-xks-ca\") pod \"c0a12187-4302-46eb-b9b5-00b18f1c8171\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") "
Oct 29 23:33:39.700479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.700481    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-xks-ca\") pod \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") "
Oct 29 23:33:39.700967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.700943    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-openshift-ca\") pod \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") "
Oct 29 23:33:39.701070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.700982    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-openshift-ca\") pod \"c0a12187-4302-46eb-b9b5-00b18f1c8171\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") "
Oct 29 23:33:39.701282 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.701253    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-kube-api-access-99gkp\" (UniqueName: \"kubernetes.io/projected/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-kube-api-access-99gkp\") pod \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\" (UID: \"ae7340bf-2bf3-4cb0-a9d8-889850892bd2\") "
Oct 29 23:33:39.701346 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.701331    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a12187-4302-46eb-b9b5-00b18f1c8171-kube-api-access-hpb24\" (UniqueName: \"kubernetes.io/projected/c0a12187-4302-46eb-b9b5-00b18f1c8171-kube-api-access-hpb24\") pod \"c0a12187-4302-46eb-b9b5-00b18f1c8171\" (UID: \"c0a12187-4302-46eb-b9b5-00b18f1c8171\") "
Oct 29 23:33:39.701398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.701348    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-openshift-ca" pod "ae7340bf-2bf3-4cb0-a9d8-889850892bd2" (UID: "ae7340bf-2bf3-4cb0-a9d8-889850892bd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:39.701742 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.701441    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-openshift-ca" pod "c0a12187-4302-46eb-b9b5-00b18f1c8171" (UID: "c0a12187-4302-46eb-b9b5-00b18f1c8171"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:39.702023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.702000    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:39.702112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.702028    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:39.703790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.703766    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-kube-api-access-99gkp" pod "ae7340bf-2bf3-4cb0-a9d8-889850892bd2" (UID: "ae7340bf-2bf3-4cb0-a9d8-889850892bd2"). InnerVolumeSpecName "kube-api-access-99gkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:39.704339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.704307    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-xks-ca" pod "ae7340bf-2bf3-4cb0-a9d8-889850892bd2" (UID: "ae7340bf-2bf3-4cb0-a9d8-889850892bd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:39.705576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.705555    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a12187-4302-46eb-b9b5-00b18f1c8171-kube-api-access-hpb24" pod "c0a12187-4302-46eb-b9b5-00b18f1c8171" (UID: "c0a12187-4302-46eb-b9b5-00b18f1c8171"). InnerVolumeSpecName "kube-api-access-hpb24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:39.706125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.706105    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-xks-ca" pod "c0a12187-4302-46eb-b9b5-00b18f1c8171" (UID: "c0a12187-4302-46eb-b9b5-00b18f1c8171"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:33:39.746857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.746772    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"]
Oct 29 23:33:39.747536 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.747511    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ae7340bf-2bf3-4cb0-a9d8-889850892bd2" containerName="revoke-keys"
Oct 29 23:33:39.747536 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.747532    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7340bf-2bf3-4cb0-a9d8-889850892bd2" containerName="revoke-keys"
Oct 29 23:33:39.747703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.747559    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c0a12187-4302-46eb-b9b5-00b18f1c8171" containerName="revoke-keys"
Oct 29 23:33:39.747703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.747566    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a12187-4302-46eb-b9b5-00b18f1c8171" containerName="revoke-keys"
Oct 29 23:33:39.747703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.747678    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7340bf-2bf3-4cb0-a9d8-889850892bd2" containerName="revoke-keys"
Oct 29 23:33:39.747703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.747694    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a12187-4302-46eb-b9b5-00b18f1c8171" containerName="revoke-keys"
Oct 29 23:33:39.751706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.751683    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.785651 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.781621    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"]
Oct 29 23:33:39.803281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803242    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.803432 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803351    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.803472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803429    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.803526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803481    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqnb\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-kube-api-access-ltqnb\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.803526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803517    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.803644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803614    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.803710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803647    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.803832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.803808    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.804124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.804068    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.804281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.804266    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a12187-4302-46eb-b9b5-00b18f1c8171-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:39.804316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.804288    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:39.804316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.804306    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99gkp\" (UniqueName: \"kubernetes.io/projected/ae7340bf-2bf3-4cb0-a9d8-889850892bd2-kube-api-access-99gkp\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:39.804383 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.804323    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpb24\" (UniqueName: \"kubernetes.io/projected/c0a12187-4302-46eb-b9b5-00b18f1c8171-kube-api-access-hpb24\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:39.905296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905266    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.905399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905337    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.905482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905424    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905490    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905528    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqnb\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-kube-api-access-ltqnb\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905581    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905683    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905702    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-certs\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905709    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.905857    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.906140    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-credential-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.906508    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-data\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.907578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.907320    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-socket\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.908385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.907681    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istiod-ca-cert\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.909234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.909200    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"]
Oct 29 23:33:39.909543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.909260    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-envoy\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.909543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.909331    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-podinfo\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.911982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.911927    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:39.912096 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.912032    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:39.912096 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.912082    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:39.933252 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:39.933218    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:33:39.938934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.938910    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-token\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:39.939074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:39.939054    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqnb\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-kube-api-access-ltqnb\") pod \"e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") " pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:40.069869 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.069824    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:40.454816 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:33:40.454779    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e123f70_499b_4e8d_a0ea_3655e76e94a1.slice/crio-abc1ed5827436bca69dc885e80c187d6813943057f77ca8d722a8d95ad7e87b3 WatchSource:0}: Error finding container abc1ed5827436bca69dc885e80c187d6813943057f77ca8d722a8d95ad7e87b3: Status 404 returned error can't find the container with id abc1ed5827436bca69dc885e80c187d6813943057f77ca8d722a8d95ad7e87b3
Oct 29 23:33:40.468738 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.468704    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"]
Oct 29 23:33:40.476227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.476195    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph" event={"ID":"c0a12187-4302-46eb-b9b5-00b18f1c8171","Type":"ContainerDied","Data":"5010612cd050084f5af0c86e8381c0f1140e912e9a0b472b1bf39a8f6655bcf5"}
Oct 29 23:33:40.476386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.476257    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5010612cd050084f5af0c86e8381c0f1140e912e9a0b472b1bf39a8f6655bcf5"
Oct 29 23:33:40.476386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.476209    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"
Oct 29 23:33:40.477236 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.477195    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:40.477808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.477779    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:40.478050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.477978    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:33:40.478620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.478594    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" event={"ID":"f1138d3a-a1ba-4384-8521-920fc1b1339a","Type":"ContainerStarted","Data":"cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463"}
Oct 29 23:33:40.478775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.478651    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" event={"ID":"f1138d3a-a1ba-4384-8521-920fc1b1339a","Type":"ContainerStarted","Data":"54f0b37aa4282d41e61787f9ac15a2bec568cda828976b5ef584377a54b343ee"}
Oct 29 23:33:40.481098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.481064    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4" event={"ID":"ae7340bf-2bf3-4cb0-a9d8-889850892bd2","Type":"ContainerDied","Data":"6d343be304865bb92490a1d5838a3c50691621b07356c94f8cb0cf893924ecce"}
Oct 29 23:33:40.481195 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.481112    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d343be304865bb92490a1d5838a3c50691621b07356c94f8cb0cf893924ecce"
Oct 29 23:33:40.481195 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.481132    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"
Oct 29 23:33:40.539032 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.538972    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podStartSLOduration=1.538953479 podStartE2EDuration="1.538953479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:40.537357035 +0000 UTC m=+1198.641617479" watchObservedRunningTime="2026-10-29 23:33:40.538953479 +0000 UTC m=+1198.643213865"
Oct 29 23:33:40.547345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.547305    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:40.549848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.549819    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:40.550056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:40.550033    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:41.558927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:41.553124    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" event={"ID":"7e123f70-499b-4e8d-a0ea-3655e76e94a1","Type":"ContainerStarted","Data":"a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d"}
Oct 29 23:33:41.558927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:41.553252    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" event={"ID":"7e123f70-499b-4e8d-a0ea-3655e76e94a1","Type":"ContainerStarted","Data":"abc1ed5827436bca69dc885e80c187d6813943057f77ca8d722a8d95ad7e87b3"}
Oct 29 23:33:41.558927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:41.554581    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:41.558927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:41.554636    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:41.657588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:41.653539    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podStartSLOduration=2.65351751 podStartE2EDuration="2.65351751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:33:41.635412302 +0000 UTC m=+1199.739672690" watchObservedRunningTime="2026-10-29 23:33:41.65351751 +0000 UTC m=+1199.757777897"
Oct 29 23:33:42.054253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.052997    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded"
Oct 29 23:33:42.106016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.102976    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:42.106016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.103045    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:42.135310 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.135272    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:42.358961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.325256    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:42.400905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.385995    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:42.400905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.386481    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/health\": context deadline exceeded"
Oct 29 23:33:42.462787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.439960    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:33:42.462787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.449244    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:42.464556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.464512    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:42.566906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.566791    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:42.566906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:42.566853    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:43.074479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.074446    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:43.074771 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.074743    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:43.104232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.104191    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:43.104936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.104915    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:33:43.496947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.412433    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:43.496947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.444607    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:33:43.496947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.446683    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded"
Oct 29 23:33:43.496947 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.447145    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:33:43.541984 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.497779    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:33:43.541984 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.498455    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:33:43.568916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.547842    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:43.583908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.575488    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:33:43.667921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.591236    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:33:43.667921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.591298    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:43.667921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.662803    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:43.667921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:43.662912    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:33:44.160957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.137588    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:44.328320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.328195    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:44.328641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.328608    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:44.393600 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.393527    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": context deadline exceeded"
Oct 29 23:33:44.421303 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.421256    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 29 23:33:44.426185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.426138    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 29 23:33:44.612680 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.612574    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:44.653255 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.653201    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:44.678922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.678850    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:33:44.821393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:44.819443    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:33:45.097916 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:33:45.092366    6203 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.043s"
Oct 29 23:33:45.295908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.269031    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded"
Oct 29 23:33:45.295908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.270960    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:45.295908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.271025    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:45.295908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.271124    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" podUID="e861dc13-1a62-4d01-94c4-e6e3288416de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:45.295908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.278144    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:45.295908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.278206    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:45.296540 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.149189    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Oct 29 23:33:45.384144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.384101    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" podUID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:33:45.494902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.494599    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:45.569782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.569679    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:45.570105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.570050    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:45.643901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.631947    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:33:45.668031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.667979    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:45.806419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.760121    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded"
Oct 29 23:33:45.822110 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.822023    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:33:45.888052 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.888000    6203 patch_prober.go:29] interesting pod/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 29 23:33:45.888358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:45.888329    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 29 23:33:46.198903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.198100    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:46.198903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.198280    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:46.271821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.271755    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:33:46.517916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.437803    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:33:46.517916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.437896    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:33:46.601926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.600471    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:46.601926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.600532    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:46.606320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.603284    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:33:46.606320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.603416    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:33:46.608838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.607369    6203 generic.go:356] "Generic (PLEG): container finished" podID="44ee719e-10b5-4394-a7d0-6820eda97835" containerID="6d130e084323b11dc8df3a5b08d8c38a1e1d50e0a102e786bc1ce9070513ca7c" exitCode=0
Oct 29 23:33:46.608838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.607573    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" event={"ID":"44ee719e-10b5-4394-a7d0-6820eda97835","Type":"ContainerDied","Data":"6d130e084323b11dc8df3a5b08d8c38a1e1d50e0a102e786bc1ce9070513ca7c"}
Oct 29 23:33:46.608838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.607840    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:33:46.608838 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.608075    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:33:46.617220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.617190    6203 generic.go:356] "Generic (PLEG): container finished" podID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerID="19f8bcda38c8df72ad65b841442d0fbaec40b3b3c43a5a893d03e6d31889175b" exitCode=0
Oct 29 23:33:46.617411 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.617394    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" event={"ID":"7394c95c-d268-48c5-9c70-f065b1a8a536","Type":"ContainerDied","Data":"19f8bcda38c8df72ad65b841442d0fbaec40b3b3c43a5a893d03e6d31889175b"}
Oct 29 23:33:46.620918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.619718    6203 generic.go:356] "Generic (PLEG): container finished" podID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerID="757122e7823460841fca55431016fde9e927cb96416b57b858f94ecb6b65aa13" exitCode=0
Oct 29 23:33:46.620918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.620106    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" event={"ID":"cd20e7cd-7216-4eb4-8880-6f882e08aead","Type":"ContainerDied","Data":"757122e7823460841fca55431016fde9e927cb96416b57b858f94ecb6b65aa13"}
Oct 29 23:33:46.722333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.722255    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810404    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-envoy\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810456    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-podinfo\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810602    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-token\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-token\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810707    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-data\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810750    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-socket\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810814    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-certs\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810932    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-kube-api-access-fpbnt\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-kube-api-access-fpbnt\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.810979    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-credential-socket\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.811005    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd20e7cd-7216-4eb4-8880-6f882e08aead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd20e7cd-7216-4eb4-8880-6f882e08aead-istiod-ca-cert\") pod \"cd20e7cd-7216-4eb4-8880-6f882e08aead\" (UID: \"cd20e7cd-7216-4eb4-8880-6f882e08aead\") "
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.812567    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd20e7cd-7216-4eb4-8880-6f882e08aead-istiod-ca-cert" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.813918    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-socket" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.814636    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-certs" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.814804    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-credential-socket" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.817928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.816245    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-data" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.883696 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.837010    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-token" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:46.893142 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.893084    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-podinfo" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:46.902190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.898347    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-kube-api-access-fpbnt" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "kube-api-access-fpbnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:46.902190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.902010    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-envoy" pod "cd20e7cd-7216-4eb4-8880-6f882e08aead" (UID: "cd20e7cd-7216-4eb4-8880-6f882e08aead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:46.912811 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912782    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912870    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpbnt\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-kube-api-access-fpbnt\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912925    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912941    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd20e7cd-7216-4eb4-8880-6f882e08aead-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912956    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912969    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912981    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.912995    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.913012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.913007    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd20e7cd-7216-4eb4-8880-6f882e08aead-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:46.930018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:46.927155    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:33:47.018731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.018638    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-envoy\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.019508 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.019486    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-data\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.020003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.019928    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-data" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.020548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.020528    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-podinfo\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.021172 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.021142    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-credential-socket" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.021354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.020868    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-credential-socket\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.021512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.021491    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7394c95c-d268-48c5-9c70-f065b1a8a536-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7394c95c-d268-48c5-9c70-f065b1a8a536-istiod-ca-cert\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.022078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.021958    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7394c95c-d268-48c5-9c70-f065b1a8a536-istiod-ca-cert" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:47.022556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.022033    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-kube-api-access-5h7lf\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-kube-api-access-5h7lf\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.023055 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.023025    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-socket" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.023230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.023211    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-socket\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.023381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.023368    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-certs\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.024103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.023662    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-certs" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.024240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.024063    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-podinfo" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:47.024400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.024382    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-token\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-token\") pod \"7394c95c-d268-48c5-9c70-f065b1a8a536\" (UID: \"7394c95c-d268-48c5-9c70-f065b1a8a536\") "
Oct 29 23:33:47.025510 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.025486    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.025707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.025683    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.025842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.025826    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.026073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.026057    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7394c95c-d268-48c5-9c70-f065b1a8a536-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.026144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.026131    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.026396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.026150    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.026700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.026673    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-envoy" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.031948 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.027865    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-token" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:47.053712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.038276    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-kube-api-access-5h7lf" pod "7394c95c-d268-48c5-9c70-f065b1a8a536" (UID: "7394c95c-d268-48c5-9c70-f065b1a8a536"). InnerVolumeSpecName "kube-api-access-5h7lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:47.101942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.100137    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:47.101942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.100191    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:47.119597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.119569    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:33:47.133908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.128116    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.133908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.128149    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7394c95c-d268-48c5-9c70-f065b1a8a536-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.133908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.128166    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7lf\" (UniqueName: \"kubernetes.io/projected/7394c95c-d268-48c5-9c70-f065b1a8a536-kube-api-access-5h7lf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.231412 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.231367    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-istio-token\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-istio-token\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.231664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.231644    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-envoy\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.231805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.231790    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-kube-api-access-gpn7m\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-kube-api-access-gpn7m\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.253902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.231961    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-certs\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.254232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.254201    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-data\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.254459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.254442    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44ee719e-10b5-4394-a7d0-6820eda97835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ee719e-10b5-4394-a7d0-6820eda97835-istio-podinfo\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.254638 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.254623    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-socket\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.254840 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.254824    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-credential-socket\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.255019 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.254988    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ee719e-10b5-4394-a7d0-6820eda97835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ee719e-10b5-4394-a7d0-6820eda97835-istiod-ca-cert\") pod \"44ee719e-10b5-4394-a7d0-6820eda97835\" (UID: \"44ee719e-10b5-4394-a7d0-6820eda97835\") "
Oct 29 23:33:47.256173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.239924    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-certs" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.256314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.256145    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ee719e-10b5-4394-a7d0-6820eda97835-istiod-ca-cert" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:47.256683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.256663    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-data" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.260897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.257445    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-credential-socket" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.260897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.257793    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-socket" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.291911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.291820    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-istio-token" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:47.293149 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.291848    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-kube-api-access-gpn7m" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "kube-api-access-gpn7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:47.293325 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.292002    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44ee719e-10b5-4394-a7d0-6820eda97835-istio-podinfo" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:33:47.293449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.292374    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-envoy" pod "44ee719e-10b5-4394-a7d0-6820eda97835" (UID: "44ee719e-10b5-4394-a7d0-6820eda97835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:33:47.382940 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.373015    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ee719e-10b5-4394-a7d0-6820eda97835-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383207 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383177    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383301    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383413 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383401    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ee719e-10b5-4394-a7d0-6820eda97835-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383517    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383608    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383711 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383700    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpn7m\" (UniqueName: \"kubernetes.io/projected/44ee719e-10b5-4394-a7d0-6820eda97835-kube-api-access-gpn7m\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383798 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383786    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.383892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.383864    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ee719e-10b5-4394-a7d0-6820eda97835-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:47.549958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.549394    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:47.549958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.549459    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:47.666901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.665483    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45" event={"ID":"cd20e7cd-7216-4eb4-8880-6f882e08aead","Type":"ContainerDied","Data":"470de3ed7dae3a05bf0f6d779f889c3e2bdc6d0f7e25c9d87968b041de63b27c"}
Oct 29 23:33:47.666901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.665562    6203 scope.go:122] "RemoveContainer" containerID="757122e7823460841fca55431016fde9e927cb96416b57b858f94ecb6b65aa13"
Oct 29 23:33:47.666901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.665751    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"
Oct 29 23:33:47.685902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.682370    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz" event={"ID":"44ee719e-10b5-4394-a7d0-6820eda97835","Type":"ContainerDied","Data":"f4315441ab9562127ee4e87a6aa7788c33687d6094e940e005131a94be788400"}
Oct 29 23:33:47.685902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.682493    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"
Oct 29 23:33:47.691780 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.691736    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt" event={"ID":"7394c95c-d268-48c5-9c70-f065b1a8a536","Type":"ContainerDied","Data":"6c6f4db9610226070228b36feaf0c37c14d735a3749cecb3ea61ea045ae58657"}
Oct 29 23:33:47.691956 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.691864    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"
Oct 29 23:33:47.708619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.708465    6203 scope.go:122] "RemoveContainer" containerID="6d130e084323b11dc8df3a5b08d8c38a1e1d50e0a102e786bc1ce9070513ca7c"
Oct 29 23:33:47.720081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.720054    6203 scope.go:122] "RemoveContainer" containerID="19f8bcda38c8df72ad65b841442d0fbaec40b3b3c43a5a893d03e6d31889175b"
Oct 29 23:33:47.821008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.819319    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"]
Oct 29 23:33:47.826764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.826730    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-jd2wz"]
Oct 29 23:33:47.895924 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.894066    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"]
Oct 29 23:33:47.900228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.900204    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-lwqbt"]
Oct 29 23:33:47.914698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.914667    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"]
Oct 29 23:33:47.920112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:47.920087    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-92f6b3-openshift-default-64f877b587-4fk45"]
Oct 29 23:33:48.019364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.019334    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" path="/var/lib/kubelet/pods/44ee719e-10b5-4394-a7d0-6820eda97835/volumes"
Oct 29 23:33:48.019863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.019841    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" path="/var/lib/kubelet/pods/7394c95c-d268-48c5-9c70-f065b1a8a536/volumes"
Oct 29 23:33:48.020323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.020309    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" path="/var/lib/kubelet/pods/cd20e7cd-7216-4eb4-8880-6f882e08aead/volumes"
Oct 29 23:33:48.086337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.086220    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:48.086337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.086285    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:48.593511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.593469    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:48.593695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.593533    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:48.703770 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.703731    6203 generic.go:356] "Generic (PLEG): container finished" podID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" containerID="30129b99ffea6d63c32d481a9ce4e7d7ba2aa6204cb3ecd319fa9bc31971e2b5" exitCode=0
Oct 29 23:33:48.703985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.703790    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" event={"ID":"65ffd545-fdf2-43d3-a726-a6fe4099d02e","Type":"ContainerDied","Data":"30129b99ffea6d63c32d481a9ce4e7d7ba2aa6204cb3ecd319fa9bc31971e2b5"}
Oct 29 23:33:48.707193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.707167    6203 generic.go:356] "Generic (PLEG): container finished" podID="e861dc13-1a62-4d01-94c4-e6e3288416de" containerID="f8c174fb0a1ec584c1a6dc060330e42ebec8b9ccb87abee027dbeb05e333e8ea" exitCode=0
Oct 29 23:33:48.707323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.707200    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" event={"ID":"e861dc13-1a62-4d01-94c4-e6e3288416de","Type":"ContainerDied","Data":"f8c174fb0a1ec584c1a6dc060330e42ebec8b9ccb87abee027dbeb05e333e8ea"}
Oct 29 23:33:48.707323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.707257    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw" event={"ID":"e861dc13-1a62-4d01-94c4-e6e3288416de","Type":"ContainerDied","Data":"aa7e6d15817f04dc4c8b8769933e8fdc8f6b5f5e3a0efc3efbad4440851a09c3"}
Oct 29 23:33:48.707323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.707279    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa7e6d15817f04dc4c8b8769933e8fdc8f6b5f5e3a0efc3efbad4440851a09c3"
Oct 29 23:33:48.723398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.723367    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:33:48.809177 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.809146    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e861dc13-1a62-4d01-94c4-e6e3288416de-kube-api-access-6wdlr\" (UniqueName: \"kubernetes.io/projected/e861dc13-1a62-4d01-94c4-e6e3288416de-kube-api-access-6wdlr\") pod \"e861dc13-1a62-4d01-94c4-e6e3288416de\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") "
Oct 29 23:33:48.809667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.809642    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e861dc13-1a62-4d01-94c4-e6e3288416de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e861dc13-1a62-4d01-94c4-e6e3288416de-plugins-config-volume\") pod \"e861dc13-1a62-4d01-94c4-e6e3288416de\" (UID: \"e861dc13-1a62-4d01-94c4-e6e3288416de\") "
Oct 29 23:33:48.810390 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.810366    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e861dc13-1a62-4d01-94c4-e6e3288416de-plugins-config-volume" pod "e861dc13-1a62-4d01-94c4-e6e3288416de" (UID: "e861dc13-1a62-4d01-94c4-e6e3288416de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:48.814100 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.814075    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e861dc13-1a62-4d01-94c4-e6e3288416de-kube-api-access-6wdlr" pod "e861dc13-1a62-4d01-94c4-e6e3288416de" (UID: "e861dc13-1a62-4d01-94c4-e6e3288416de"). InnerVolumeSpecName "kube-api-access-6wdlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:48.888424 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.888338    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:33:48.912126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.912100    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wdlr\" (UniqueName: \"kubernetes.io/projected/e861dc13-1a62-4d01-94c4-e6e3288416de-kube-api-access-6wdlr\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:48.912126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:48.912123    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e861dc13-1a62-4d01-94c4-e6e3288416de-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:49.012828 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.012784    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ffd545-fdf2-43d3-a726-a6fe4099d02e-kube-api-access-qjrjw\" (UniqueName: \"kubernetes.io/projected/65ffd545-fdf2-43d3-a726-a6fe4099d02e-kube-api-access-qjrjw\") pod \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") "
Oct 29 23:33:49.013041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.012945    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ffd545-fdf2-43d3-a726-a6fe4099d02e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ffd545-fdf2-43d3-a726-a6fe4099d02e-plugins-config-volume\") pod \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\" (UID: \"65ffd545-fdf2-43d3-a726-a6fe4099d02e\") "
Oct 29 23:33:49.013402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.013377    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ffd545-fdf2-43d3-a726-a6fe4099d02e-plugins-config-volume" pod "65ffd545-fdf2-43d3-a726-a6fe4099d02e" (UID: "65ffd545-fdf2-43d3-a726-a6fe4099d02e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:33:49.015152 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.015122    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ffd545-fdf2-43d3-a726-a6fe4099d02e-kube-api-access-qjrjw" pod "65ffd545-fdf2-43d3-a726-a6fe4099d02e" (UID: "65ffd545-fdf2-43d3-a726-a6fe4099d02e"). InnerVolumeSpecName "kube-api-access-qjrjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:33:49.078925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.071282    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:49.078925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.071348    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:49.115144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.115070    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65ffd545-fdf2-43d3-a726-a6fe4099d02e-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:49.115144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.115111    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjrjw\" (UniqueName: \"kubernetes.io/projected/65ffd545-fdf2-43d3-a726-a6fe4099d02e-kube-api-access-qjrjw\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:33:49.555090 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.550856    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 29 23:33:49.555090 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.550946    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 29 23:33:49.718070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.718032    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"
Oct 29 23:33:49.718273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.718066    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl" event={"ID":"65ffd545-fdf2-43d3-a726-a6fe4099d02e","Type":"ContainerDied","Data":"d26aa214a9e26d2bdc336d827f4a4483ab79dafcb7cc3b5ae9f83eb4cbe5b72c"}
Oct 29 23:33:49.718273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.718135    6203 scope.go:122] "RemoveContainer" containerID="30129b99ffea6d63c32d481a9ce4e7d7ba2aa6204cb3ecd319fa9bc31971e2b5"
Oct 29 23:33:49.718273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.718190    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"
Oct 29 23:33:49.816622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.816527    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"]
Oct 29 23:33:49.854903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.851742    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b978a0-6bb99969b4-mxtvw"]
Oct 29 23:33:49.875823 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.875790    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"]
Oct 29 23:33:49.880239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:49.880208    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b978a0-d865657f5-f84kl"]
Oct 29 23:33:50.019924 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:50.019865    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" path="/var/lib/kubelet/pods/65ffd545-fdf2-43d3-a726-a6fe4099d02e/volumes"
Oct 29 23:33:50.020386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:50.020371    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e861dc13-1a62-4d01-94c4-e6e3288416de" path="/var/lib/kubelet/pods/e861dc13-1a62-4d01-94c4-e6e3288416de/volumes"
Oct 29 23:33:50.070459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:50.070365    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 29 23:33:50.070459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:50.070418    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 29 23:33:51.024987 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:51.024954    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:51.025534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:51.025511    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:51.026842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:51.026538    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:33:51.272861 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:51.272825    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:51.273692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:51.273667    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:51.274863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:51.274840    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:33:58.642051 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:58.642018    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"]
Oct 29 23:33:58.656749 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:33:58.656715    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b978a0-w5t9v"]
Oct 29 23:34:00.019873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:00.019838    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5b5914-9d4e-45c9-bbb8-7e0eed929184" path="/var/lib/kubelet/pods/9a5b5914-9d4e-45c9-bbb8-7e0eed929184/volumes"
Oct 29 23:34:01.930834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.930787    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"]
Oct 29 23:34:01.931604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931579    6203 cpu_manager.go:414] "containerMap: removing container" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy"
Oct 29 23:34:01.931604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931605    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931635    6203 cpu_manager.go:414] "containerMap: removing container" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931643    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931673    6203 cpu_manager.go:414] "containerMap: removing container" podUID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" containerName="payload-pre-processing"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931683    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" containerName="payload-pre-processing"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931705    6203 cpu_manager.go:414] "containerMap: removing container" podUID="e861dc13-1a62-4d01-94c4-e6e3288416de" containerName="payload-processing"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931712    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e861dc13-1a62-4d01-94c4-e6e3288416de" containerName="payload-processing"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931734    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy"
Oct 29 23:34:01.931779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931742    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy"
Oct 29 23:34:01.932197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931927    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd20e7cd-7216-4eb4-8880-6f882e08aead" containerName="istio-proxy"
Oct 29 23:34:01.932197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931951    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ffd545-fdf2-43d3-a726-a6fe4099d02e" containerName="payload-pre-processing"
Oct 29 23:34:01.932197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931963    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="e861dc13-1a62-4d01-94c4-e6e3288416de" containerName="payload-processing"
Oct 29 23:34:01.932197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931976    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7394c95c-d268-48c5-9c70-f065b1a8a536" containerName="istio-proxy"
Oct 29 23:34:01.932197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.931990    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ee719e-10b5-4394-a7d0-6820eda97835" containerName="istio-proxy"
Oct 29 23:34:01.935730 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.935709    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:01.956685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.956655    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"]
Oct 29 23:34:01.965521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.965488    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:34:01.965687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:01.965547    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:34:02.003676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.003639    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.003862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.003707    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dsll\" (UniqueName: \"kubernetes.io/projected/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-kube-api-access-2dsll\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.003931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.003892    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.043699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.043661    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:34:02.043699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.043692    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:34:02.105331 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.105291    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.105482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.105435    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.105580 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.105530    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dsll\" (UniqueName: \"kubernetes.io/projected/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-kube-api-access-2dsll\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.106157 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.106123    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.108029 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.108008    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.131709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.131681    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dsll\" (UniqueName: \"kubernetes.io/projected/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-kube-api-access-2dsll\") pod \"maas-api-revoke-sub-5584c3af-6b4dc\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.249301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.249208    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:02.560327 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:02.560295    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff5ac49_1de5_4e36_abe0_ee91b4ce1ebd.slice/crio-378649ba60067b5ee9d80519c8f1d10c95f22206472315e4ff9ac1ded6aabf17 WatchSource:0}: Error finding container 378649ba60067b5ee9d80519c8f1d10c95f22206472315e4ff9ac1ded6aabf17: Status 404 returned error can't find the container with id 378649ba60067b5ee9d80519c8f1d10c95f22206472315e4ff9ac1ded6aabf17
Oct 29 23:34:02.560991 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.560968    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"]
Oct 29 23:34:02.831869 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:02.831837    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc" event={"ID":"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd","Type":"ContainerStarted","Data":"378649ba60067b5ee9d80519c8f1d10c95f22206472315e4ff9ac1ded6aabf17"}
Oct 29 23:34:03.846438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:03.846344    6203 generic.go:356] "Generic (PLEG): container finished" podID="fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" containerID="bc85759c212353b3d9aaca72c41cf58f95b7cbaab2aa4d703b23ffde953c71c0" exitCode=0
Oct 29 23:34:03.846832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:03.846428    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc" event={"ID":"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd","Type":"ContainerDied","Data":"bc85759c212353b3d9aaca72c41cf58f95b7cbaab2aa4d703b23ffde953c71c0"}
Oct 29 23:34:05.025704 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.025676    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:05.050706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.050679    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-kube-api-access-2dsll\" (UniqueName: \"kubernetes.io/projected/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-kube-api-access-2dsll\") pod \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") "
Oct 29 23:34:05.050834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.050737    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-xks-ca\") pod \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") "
Oct 29 23:34:05.050905 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.050864    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-openshift-ca\") pod \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\" (UID: \"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd\") "
Oct 29 23:34:05.051186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.051158    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-openshift-ca" pod "fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" (UID: "fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:05.051317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.051274    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:05.052899 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.052865    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-xks-ca" pod "fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" (UID: "fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:05.053047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.053026    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-kube-api-access-2dsll" pod "fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" (UID: "fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd"). InnerVolumeSpecName "kube-api-access-2dsll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:05.152801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.152716    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dsll\" (UniqueName: \"kubernetes.io/projected/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-kube-api-access-2dsll\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:05.152801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.152748    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:05.819607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.819573    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"]
Oct 29 23:34:05.820117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.820086    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy" containerID="cri-o://b2a613f843cd578eed854a2977286a5c4654b4bbef9846d487d5d4f255149daf" gracePeriod=30
Oct 29 23:34:05.834084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.834023    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"]
Oct 29 23:34:05.834292 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.834266    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy" containerID="cri-o://24b9191d859b432356f985f2404986cdd79938be01c274b4bdeec51cbc39aa3e" gracePeriod=30
Oct 29 23:34:05.845989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.845961    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"]
Oct 29 23:34:05.846159 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.846141    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy" containerID="cri-o://a35f185a27425db3d6d79f009c05481df89e94523dfc74119916513fe662c2d1" gracePeriod=30
Oct 29 23:34:05.863438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.863408    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"
Oct 29 23:34:05.863581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.863437    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc" event={"ID":"fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd","Type":"ContainerDied","Data":"378649ba60067b5ee9d80519c8f1d10c95f22206472315e4ff9ac1ded6aabf17"}
Oct 29 23:34:05.863581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:05.863488    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378649ba60067b5ee9d80519c8f1d10c95f22206472315e4ff9ac1ded6aabf17"
Oct 29 23:34:09.288914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:09.286967    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:09.337030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:09.336975    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:09.429925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:09.429861    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:09.634151 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:09.634052    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 29 23:34:09.686902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:09.682977    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:09.818783 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:09.818739    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded"
Oct 29 23:34:10.290016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.289968    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:10.291355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.291329    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:34:10.343173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.343132    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:34:10.343788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.343761    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:34:10.355907 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:34:10.346213    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:34:10.575955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.564980    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:10.578203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.578175    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:34:10.646100 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.646059    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:10.849962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.742288    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:10.849962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.836392    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:10.923779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.890705    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:34:10.923779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.890778    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:34:10.923779 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:10.890826    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:34:11.213105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:11.143058    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:11.342224 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:11.291161    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:11.369663 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:11.350381    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:11.630959 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:11.629023    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:11.817903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:11.809814    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:11.906007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:11.880761    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:11.953497 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:11.953452    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:12.297270 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.297226    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:12.436010 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.416591    6203 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:12.468973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.461275    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 29 23:34:12.665956 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.654354    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:34:12.679141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.679095    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:12.708901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.694020    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:34:12.744090 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.744051    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:34:12.821997 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.821950    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk" podUID="1e8aa778-5206-47f8-8379-03976219cdcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:34:12.830944 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.830771    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:34:12.833469 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.833447    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:34:12.834650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.834629    6203 generic.go:356] "Generic (PLEG): container finished" podID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerID="24b9191d859b432356f985f2404986cdd79938be01c274b4bdeec51cbc39aa3e" exitCode=0
Oct 29 23:34:12.834973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.834954    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" event={"ID":"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7","Type":"ContainerDied","Data":"24b9191d859b432356f985f2404986cdd79938be01c274b4bdeec51cbc39aa3e"}
Oct 29 23:34:12.837531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.837511    6203 generic.go:356] "Generic (PLEG): container finished" podID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerID="b2a613f843cd578eed854a2977286a5c4654b4bbef9846d487d5d4f255149daf" exitCode=0
Oct 29 23:34:12.837787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.837770    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" event={"ID":"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba","Type":"ContainerDied","Data":"b2a613f843cd578eed854a2977286a5c4654b4bbef9846d487d5d4f255149daf"}
Oct 29 23:34:12.841772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.841754    6203 generic.go:356] "Generic (PLEG): container finished" podID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerID="a35f185a27425db3d6d79f009c05481df89e94523dfc74119916513fe662c2d1" exitCode=0
Oct 29 23:34:12.842007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.841907    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" event={"ID":"884c8036-da5c-4e3f-a7c1-bf149f46356b","Type":"ContainerDied","Data":"a35f185a27425db3d6d79f009c05481df89e94523dfc74119916513fe662c2d1"}
Oct 29 23:34:12.842140 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.842130    6203 scope.go:122] "RemoveContainer" containerID="be2e67945cae744f22f764c36b8ccf12d01bfe44bcc62ff255a9490968902026"
Oct 29 23:34:12.851285 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.851265    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk"
Oct 29 23:34:12.996447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:12.996352    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:34:13.030338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030308    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884c8036-da5c-4e3f-a7c1-bf149f46356b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8036-da5c-4e3f-a7c1-bf149f46356b-istiod-ca-cert\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.030542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030407    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-certs\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.030542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030442    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-credential-socket\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.030542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030521    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-token\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-token\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.030703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030565    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-socket\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.030703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030621    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-kube-api-access-6l6wt\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-kube-api-access-6l6wt\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.030703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030658    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-envoy\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.030863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030747    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884c8036-da5c-4e3f-a7c1-bf149f46356b-istiod-ca-cert" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:13.030952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030744    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-certs" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.030952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030853    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-credential-socket" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.030952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030815    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-data\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.031114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.030948    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-podinfo\") pod \"884c8036-da5c-4e3f-a7c1-bf149f46356b\" (UID: \"884c8036-da5c-4e3f-a7c1-bf149f46356b\") "
Oct 29 23:34:13.031316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.031297    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-socket" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.032283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.032106    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-data" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.032413 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.031651    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884c8036-da5c-4e3f-a7c1-bf149f46356b-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.032413 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.032408    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.032523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.032424    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.034964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.034929    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-kube-api-access-6l6wt" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "kube-api-access-6l6wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:13.035066 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.034948    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-podinfo" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:13.035516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.035485    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-envoy" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.036313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.036294    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-token" pod "884c8036-da5c-4e3f-a7c1-bf149f46356b" (UID: "884c8036-da5c-4e3f-a7c1-bf149f46356b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:13.047383 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.047347    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"]
Oct 29 23:34:13.053248 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.053180    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:34:13.056025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.055995    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172702af-fvzxf"]
Oct 29 23:34:13.061421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.061401    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:34:13.133024 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.132997    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-envoy\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133184 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133033    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istiod-ca-cert\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133184 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133055    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-data\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133184 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133087    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-credential-socket\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133184 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133145    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-kube-api-access-njnf5\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-kube-api-access-njnf5\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133184 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133167    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-credential-socket\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133220    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-data\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133274    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-socket\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133342    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-data" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.133452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133363    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-certs\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133395    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-certs\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133421    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-kube-api-access-dz2nj\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-kube-api-access-dz2nj\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133450    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-token\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-token\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133496    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-podinfo\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133538    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-envoy\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133563    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istiod-ca-cert" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133563    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-credential-socket" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133586    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-data" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133593    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-socket" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133568    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-podinfo\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133752    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-socket\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133796    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-token\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-token\") pod \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\" (UID: \"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7\") "
Oct 29 23:34:13.133813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133803    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-certs" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.134374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.133840    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istiod-ca-cert\") pod \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\" (UID: \"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba\") "
Oct 29 23:34:13.134374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134002    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-socket" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.134374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134099    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-certs" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.134543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134444    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istiod-ca-cert" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:13.134543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134485    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-credential-socket" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.134649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134549    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6wt\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-kube-api-access-6l6wt\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.134649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134574    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.134649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134590    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.134649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134605    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.134649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134620    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.134649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134635    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.134649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134649    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.135047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134664    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.135047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134678    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.135047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134692    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.135047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134706    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.135047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134722    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884c8036-da5c-4e3f-a7c1-bf149f46356b-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.135047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134735    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.135047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.134750    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884c8036-da5c-4e3f-a7c1-bf149f46356b-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.136619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.136591    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-envoy" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.136619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.136601    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-token" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:13.136914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.136621    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-kube-api-access-dz2nj" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "kube-api-access-dz2nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:13.137210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.137177    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-podinfo" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:13.137390 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.137362    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-envoy" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:13.137458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.137384    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-podinfo" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:34:13.137856 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.137829    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-token" pod "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" (UID: "5c99ab68-24ca-4d67-9c01-ec5bfc1146e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:13.138294 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.138268    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-kube-api-access-njnf5" pod "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" (UID: "a6afa2e8-8160-4df4-ade8-9c1b4eec46ba"). InnerVolumeSpecName "kube-api-access-njnf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:13.197871 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.197801    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"]
Oct 29 23:34:13.198400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198386    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy"
Oct 29 23:34:13.198453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198401    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy"
Oct 29 23:34:13.198453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198410    6203 cpu_manager.go:414] "containerMap: removing container" podUID="fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" containerName="revoke-keys"
Oct 29 23:34:13.198453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198415    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" containerName="revoke-keys"
Oct 29 23:34:13.198453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198449    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy"
Oct 29 23:34:13.198567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198454    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy"
Oct 29 23:34:13.198567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198464    6203 cpu_manager.go:414] "containerMap: removing container" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy"
Oct 29 23:34:13.198567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198468    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy"
Oct 29 23:34:13.198567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198482    6203 cpu_manager.go:414] "containerMap: removing container" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy"
Oct 29 23:34:13.198567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198487    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy"
Oct 29 23:34:13.198567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198564    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" containerName="revoke-keys"
Oct 29 23:34:13.198755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198577    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" containerName="istio-proxy"
Oct 29 23:34:13.198755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198614    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy"
Oct 29 23:34:13.198755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198627    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" containerName="istio-proxy"
Oct 29 23:34:13.198755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.198639    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" containerName="istio-proxy"
Oct 29 23:34:13.205860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.205830    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.226773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.226714    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"]
Oct 29 23:34:13.227314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.227007    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:34:13.227314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.227054    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:34:13.227314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.227146    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:34:13.227314 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.227287    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:34:13.236350 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236325    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68cpq\" (UniqueName: \"kubernetes.io/projected/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-kube-api-access-68cpq\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.236479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236377    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.236579 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236564    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.236677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236666    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2nj\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-kube-api-access-dz2nj\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236680    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236689    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236697    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236705    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236712    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236720    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236728    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236736    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.236935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.236744    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnf5\" (UniqueName: \"kubernetes.io/projected/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba-kube-api-access-njnf5\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:13.338027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.337993    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.338226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.338116    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68cpq\" (UniqueName: \"kubernetes.io/projected/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-kube-api-access-68cpq\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.338226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.338179    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.338728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.338705    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.340382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.340355    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.364080 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.364045    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68cpq\" (UniqueName: \"kubernetes.io/projected/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-kube-api-access-68cpq\") pod \"maas-api-revoke-sub-b9cee766-q6n2f\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.531307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.531226    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:13.794064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.793986    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"]
Oct 29 23:34:13.799965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.799937    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.818374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.818346    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"]
Oct 29 23:34:13.835613 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:13.835582    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6f0891_7dae_4f01_a7b8_cd872705d6f1.slice/crio-c6a43b4bc701bb743e70c3258d270b0ec750efc8ba09334af3c7c7b7b52698e6 WatchSource:0}: Error finding container c6a43b4bc701bb743e70c3258d270b0ec750efc8ba09334af3c7c7b7b52698e6: Status 404 returned error can't find the container with id c6a43b4bc701bb743e70c3258d270b0ec750efc8ba09334af3c7c7b7b52698e6
Oct 29 23:34:13.835758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.835733    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"]
Oct 29 23:34:13.844371 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.844346    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.844492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.844434    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfh9\" (UniqueName: \"kubernetes.io/projected/619a55e9-8751-480c-a854-5b8c5421f6e5-kube-api-access-zjfh9\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.844560 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.844503    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.853061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.853031    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"
Oct 29 23:34:13.853061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.853049    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch" event={"ID":"a6afa2e8-8160-4df4-ade8-9c1b4eec46ba","Type":"ContainerDied","Data":"af80707ec924fbc08ae3e39d96738ba56fe0ec7b95c614e0099d8559b5406d3a"}
Oct 29 23:34:13.853249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.853110    6203 scope.go:122] "RemoveContainer" containerID="b2a613f843cd578eed854a2977286a5c4654b4bbef9846d487d5d4f255149daf"
Oct 29 23:34:13.854913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.854854    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"
Oct 29 23:34:13.855033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.854870    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p" event={"ID":"884c8036-da5c-4e3f-a7c1-bf149f46356b","Type":"ContainerDied","Data":"90a87d0dd9af98d4aab68bc4ac1fa29d566a081b78cc1e8be17ab8781f3033bb"}
Oct 29 23:34:13.857218 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.857194    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f" event={"ID":"2b6f0891-7dae-4f01-a7b8-cd872705d6f1","Type":"ContainerStarted","Data":"c6a43b4bc701bb743e70c3258d270b0ec750efc8ba09334af3c7c7b7b52698e6"}
Oct 29 23:34:13.859090 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.859037    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"
Oct 29 23:34:13.859090 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.859061    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx" event={"ID":"5c99ab68-24ca-4d67-9c01-ec5bfc1146e7","Type":"ContainerDied","Data":"76dd70be76c5a754b80f3d28d7ff43177e9dbe8545883105a2718755b4bf9deb"}
Oct 29 23:34:13.866836 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.866808    6203 scope.go:122] "RemoveContainer" containerID="a35f185a27425db3d6d79f009c05481df89e94523dfc74119916513fe662c2d1"
Oct 29 23:34:13.881506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.881487    6203 scope.go:122] "RemoveContainer" containerID="24b9191d859b432356f985f2404986cdd79938be01c274b4bdeec51cbc39aa3e"
Oct 29 23:34:13.945384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.945353    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfh9\" (UniqueName: \"kubernetes.io/projected/619a55e9-8751-480c-a854-5b8c5421f6e5-kube-api-access-zjfh9\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.945527 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.945427    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.945574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.945549    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.946164 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.946141    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.947680 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.947659    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.951343 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.951319    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"]
Oct 29 23:34:13.956382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.956360    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7pt4p"]
Oct 29 23:34:13.972344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.972324    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfh9\" (UniqueName: \"kubernetes.io/projected/619a55e9-8751-480c-a854-5b8c5421f6e5-kube-api-access-zjfh9\") pod \"maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:13.996446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:13.996420    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"]
Oct 29 23:34:14.000437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.000417    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-7s2dx"]
Oct 29 23:34:14.028018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.027990    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c99ab68-24ca-4d67-9c01-ec5bfc1146e7" path="/var/lib/kubelet/pods/5c99ab68-24ca-4d67-9c01-ec5bfc1146e7/volumes"
Oct 29 23:34:14.028837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.028799    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884c8036-da5c-4e3f-a7c1-bf149f46356b" path="/var/lib/kubelet/pods/884c8036-da5c-4e3f-a7c1-bf149f46356b/volumes"
Oct 29 23:34:14.029759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.029737    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0da7a58-0c4d-494a-89c6-f7be452f4095" path="/var/lib/kubelet/pods/a0da7a58-0c4d-494a-89c6-f7be452f4095/volumes"
Oct 29 23:34:14.046402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.046343    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"]
Oct 29 23:34:14.051819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.051796    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b978a0-openshift-default-5d657d9dff-ck5ch"]
Oct 29 23:34:14.114451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.114416    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:14.436215 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:14.436184    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619a55e9_8751_480c_a854_5b8c5421f6e5.slice/crio-a484192059b8cd810c0f1dedd0caa658fbc7a7421ceae75b2e832cfa0201367b WatchSource:0}: Error finding container a484192059b8cd810c0f1dedd0caa658fbc7a7421ceae75b2e832cfa0201367b: Status 404 returned error can't find the container with id a484192059b8cd810c0f1dedd0caa658fbc7a7421ceae75b2e832cfa0201367b
Oct 29 23:34:14.437452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.437414    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"]
Oct 29 23:34:14.871339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.871305    6203 generic.go:356] "Generic (PLEG): container finished" podID="2b6f0891-7dae-4f01-a7b8-cd872705d6f1" containerID="f7f2c5e94fcaa07b7f18c4180e3a81d6633441efa6bffec80e5070a44011c40c" exitCode=0
Oct 29 23:34:14.871809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.871339    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f" event={"ID":"2b6f0891-7dae-4f01-a7b8-cd872705d6f1","Type":"ContainerDied","Data":"f7f2c5e94fcaa07b7f18c4180e3a81d6633441efa6bffec80e5070a44011c40c"}
Oct 29 23:34:14.873136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.873089    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b" event={"ID":"619a55e9-8751-480c-a854-5b8c5421f6e5","Type":"ContainerStarted","Data":"c566eeaeea7a7a3ce1bbdfe0e21238af831a6c16d5bb93e6b6cfac4e7e5376df"}
Oct 29 23:34:14.873256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.873150    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b" event={"ID":"619a55e9-8751-480c-a854-5b8c5421f6e5","Type":"ContainerStarted","Data":"a484192059b8cd810c0f1dedd0caa658fbc7a7421ceae75b2e832cfa0201367b"}
Oct 29 23:34:14.965105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:14.965047    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b" podStartSLOduration=1.96503384 podStartE2EDuration="1.96503384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:14.962712645 +0000 UTC m=+1233.066973030" watchObservedRunningTime="2026-10-29 23:34:14.96503384 +0000 UTC m=+1233.069294220"
Oct 29 23:34:15.884943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:15.884909    6203 generic.go:356] "Generic (PLEG): container finished" podID="619a55e9-8751-480c-a854-5b8c5421f6e5" containerID="c566eeaeea7a7a3ce1bbdfe0e21238af831a6c16d5bb93e6b6cfac4e7e5376df" exitCode=0
Oct 29 23:34:15.885339 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:15.884997    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b" event={"ID":"619a55e9-8751-480c-a854-5b8c5421f6e5","Type":"ContainerDied","Data":"c566eeaeea7a7a3ce1bbdfe0e21238af831a6c16d5bb93e6b6cfac4e7e5376df"}
Oct 29 23:34:16.020798 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.020765    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6afa2e8-8160-4df4-ade8-9c1b4eec46ba" path="/var/lib/kubelet/pods/a6afa2e8-8160-4df4-ade8-9c1b4eec46ba/volumes"
Oct 29 23:34:16.055856 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.055826    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:16.172328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.172294    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-kube-api-access-68cpq\" (UniqueName: \"kubernetes.io/projected/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-kube-api-access-68cpq\") pod \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") "
Oct 29 23:34:16.172519 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.172426    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-openshift-ca\") pod \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") "
Oct 29 23:34:16.172519 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.172471    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-xks-ca\") pod \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\" (UID: \"2b6f0891-7dae-4f01-a7b8-cd872705d6f1\") "
Oct 29 23:34:16.172802 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.172776    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-openshift-ca" pod "2b6f0891-7dae-4f01-a7b8-cd872705d6f1" (UID: "2b6f0891-7dae-4f01-a7b8-cd872705d6f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:16.173108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.173092    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:16.174396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.174371    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-kube-api-access-68cpq" pod "2b6f0891-7dae-4f01-a7b8-cd872705d6f1" (UID: "2b6f0891-7dae-4f01-a7b8-cd872705d6f1"). InnerVolumeSpecName "kube-api-access-68cpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:16.174505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.174447    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-xks-ca" pod "2b6f0891-7dae-4f01-a7b8-cd872705d6f1" (UID: "2b6f0891-7dae-4f01-a7b8-cd872705d6f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:16.274962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.274930    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:16.274962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.274959    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68cpq\" (UniqueName: \"kubernetes.io/projected/2b6f0891-7dae-4f01-a7b8-cd872705d6f1-kube-api-access-68cpq\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:16.893234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.893196    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f" event={"ID":"2b6f0891-7dae-4f01-a7b8-cd872705d6f1","Type":"ContainerDied","Data":"c6a43b4bc701bb743e70c3258d270b0ec750efc8ba09334af3c7c7b7b52698e6"}
Oct 29 23:34:16.893712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.893247    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a43b4bc701bb743e70c3258d270b0ec750efc8ba09334af3c7c7b7b52698e6"
Oct 29 23:34:16.893712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:16.893304    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"
Oct 29 23:34:17.052363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.052336    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:17.084359 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.084331    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-openshift-ca\") pod \"619a55e9-8751-480c-a854-5b8c5421f6e5\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") "
Oct 29 23:34:17.084501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.084388    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-xks-ca\") pod \"619a55e9-8751-480c-a854-5b8c5421f6e5\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") "
Oct 29 23:34:17.084501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.084412    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619a55e9-8751-480c-a854-5b8c5421f6e5-kube-api-access-zjfh9\" (UniqueName: \"kubernetes.io/projected/619a55e9-8751-480c-a854-5b8c5421f6e5-kube-api-access-zjfh9\") pod \"619a55e9-8751-480c-a854-5b8c5421f6e5\" (UID: \"619a55e9-8751-480c-a854-5b8c5421f6e5\") "
Oct 29 23:34:17.084701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.084676    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-openshift-ca" pod "619a55e9-8751-480c-a854-5b8c5421f6e5" (UID: "619a55e9-8751-480c-a854-5b8c5421f6e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:17.085145 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.085122    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:17.086384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.086363    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-xks-ca" pod "619a55e9-8751-480c-a854-5b8c5421f6e5" (UID: "619a55e9-8751-480c-a854-5b8c5421f6e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:17.086540 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.086519    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619a55e9-8751-480c-a854-5b8c5421f6e5-kube-api-access-zjfh9" pod "619a55e9-8751-480c-a854-5b8c5421f6e5" (UID: "619a55e9-8751-480c-a854-5b8c5421f6e5"). InnerVolumeSpecName "kube-api-access-zjfh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:17.187443 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.187396    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a55e9-8751-480c-a854-5b8c5421f6e5-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:17.187443 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.187446    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfh9\" (UniqueName: \"kubernetes.io/projected/619a55e9-8751-480c-a854-5b8c5421f6e5-kube-api-access-zjfh9\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:17.903577 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.903540    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"
Oct 29 23:34:17.904100 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.903578    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b" event={"ID":"619a55e9-8751-480c-a854-5b8c5421f6e5","Type":"ContainerDied","Data":"a484192059b8cd810c0f1dedd0caa658fbc7a7421ceae75b2e832cfa0201367b"}
Oct 29 23:34:17.904100 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:17.903650    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a484192059b8cd810c0f1dedd0caa658fbc7a7421ceae75b2e832cfa0201367b"
Oct 29 23:34:18.039918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:18.039872    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"]
Oct 29 23:34:24.044347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.044308    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"]
Oct 29 23:34:24.058680 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.058651    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9ea233-blmg5"]
Oct 29 23:34:24.390330 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.390251    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"]
Oct 29 23:34:24.390819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.390806    6203 cpu_manager.go:414] "containerMap: removing container" podUID="619a55e9-8751-480c-a854-5b8c5421f6e5" containerName="revoke-keys"
Oct 29 23:34:24.390867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.390820    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619a55e9-8751-480c-a854-5b8c5421f6e5" containerName="revoke-keys"
Oct 29 23:34:24.390867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.390837    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2b6f0891-7dae-4f01-a7b8-cd872705d6f1" containerName="revoke-keys"
Oct 29 23:34:24.390867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.390842    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6f0891-7dae-4f01-a7b8-cd872705d6f1" containerName="revoke-keys"
Oct 29 23:34:24.391025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.390980    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="619a55e9-8751-480c-a854-5b8c5421f6e5" containerName="revoke-keys"
Oct 29 23:34:24.391025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.390993    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6f0891-7dae-4f01-a7b8-cd872705d6f1" containerName="revoke-keys"
Oct 29 23:34:24.395462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.395423    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.436976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.425968    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"]
Oct 29 23:34:24.482216 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482163    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktpz\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-kube-api-access-9ktpz\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482412 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482237    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482412 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482279    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482412 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482336    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482429    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482460    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482518    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482587    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.482799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.482672    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.583707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.583676    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.583919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.583718    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.583919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.583848    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.583976    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584001    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584045    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584241 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584111    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584241 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584191    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584244    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584263    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktpz\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-kube-api-access-9ktpz\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584285    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-certs\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584413    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-data\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584462    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-credential-socket\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.584629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.584539    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istiod-ca-cert\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.586468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.586441    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-envoy\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.586752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.586732    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-podinfo\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.611767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.611736    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-token\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.611941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.611924    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktpz\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-kube-api-access-9ktpz\") pod \"e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") " pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:24.715004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:24.714972    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:25.064291 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:25.064258    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0fa53c_cb5f_4552_9d6b_fc8aebea0504.slice/crio-3b764a98cfa76261424147afe9152269ecc7c3c0f415fc56969e81cdc0bd4fe7 WatchSource:0}: Error finding container 3b764a98cfa76261424147afe9152269ecc7c3c0f415fc56969e81cdc0bd4fe7: Status 404 returned error can't find the container with id 3b764a98cfa76261424147afe9152269ecc7c3c0f415fc56969e81cdc0bd4fe7
Oct 29 23:34:25.065340 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.065308    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"]
Oct 29 23:34:25.066765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.066732    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:34:25.066855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.066794    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:34:25.066855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.066823    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:34:25.102870 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.102845    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"]
Oct 29 23:34:25.110816 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.110789    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3049dd0-r849b"]
Oct 29 23:34:25.971575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.971537    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" event={"ID":"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504","Type":"ContainerStarted","Data":"5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7"}
Oct 29 23:34:25.971762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:25.971588    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" event={"ID":"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504","Type":"ContainerStarted","Data":"3b764a98cfa76261424147afe9152269ecc7c3c0f415fc56969e81cdc0bd4fe7"}
Oct 29 23:34:26.019641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:26.019611    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89242841-51c2-4bc9-977f-ed1020eb21b9" path="/var/lib/kubelet/pods/89242841-51c2-4bc9-977f-ed1020eb21b9/volumes"
Oct 29 23:34:26.020062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:26.020043    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb18175-ae67-4e52-a022-00368741f817" path="/var/lib/kubelet/pods/cbb18175-ae67-4e52-a022-00368741f817/volumes"
Oct 29 23:34:26.050902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:26.045724    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podStartSLOduration=2.045705142 podStartE2EDuration="2.045705142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:26.045000986 +0000 UTC m=+1244.149261374" watchObservedRunningTime="2026-10-29 23:34:26.045705142 +0000 UTC m=+1244.149965530"
Oct 29 23:34:26.716068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:26.716024    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:26.717191 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:26.717157    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 29 23:34:26.717344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:26.717222    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 29 23:34:27.716237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:27.715548    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 29 23:34:27.716237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:27.715631    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 29 23:34:28.716289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:28.716254    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 29 23:34:28.716706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:28.716308    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 29 23:34:29.715789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:29.715752    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 29 23:34:29.716006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:29.715819    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 29 23:34:30.716160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:30.716119    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 29 23:34:30.716593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:30.716189    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 29 23:34:31.651310 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.651268    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"]
Oct 29 23:34:31.658847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.658814    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.680643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.680610    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"]
Oct 29 23:34:31.685298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.685270    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:34:31.685468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.685353    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:34:31.685468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.685400    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:34:31.687216 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.687184    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"]
Oct 29 23:34:31.690505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.690479    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:34:31.692421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.692398    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.708937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.708912    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"]
Oct 29 23:34:31.715530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.715504    6203 patch_prober.go:29] interesting pod/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 29 23:34:31.715732 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.715704    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 29 23:34:31.814692 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.814655    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.815120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.814707    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkkf\" (UniqueName: \"kubernetes.io/projected/8b817b21-4bf2-415a-928b-cb2f20a5b995-kube-api-access-tfkkf\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.815120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.814833    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4gh4\" (UniqueName: \"kubernetes.io/projected/293eae7c-c172-4580-b4d4-5933e97ed0fc-kube-api-access-h4gh4\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.815120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.814937    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.815120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.815000    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.815257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.815117    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.916698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.916596    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gh4\" (UniqueName: \"kubernetes.io/projected/293eae7c-c172-4580-b4d4-5933e97ed0fc-kube-api-access-h4gh4\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.916871 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.916724    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.916871 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.916776    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.916994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.916936    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.917074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.917055    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.917133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.917095    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkkf\" (UniqueName: \"kubernetes.io/projected/8b817b21-4bf2-415a-928b-cb2f20a5b995-kube-api-access-tfkkf\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.917633 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.917601    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.917844 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.917818    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.919398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.919373    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.919490 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.919381    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.943938 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.943833    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkkf\" (UniqueName: \"kubernetes.io/projected/8b817b21-4bf2-415a-928b-cb2f20a5b995-kube-api-access-tfkkf\") pod \"maas-api-revoke-sub-e2d22e6e-98rdt\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:31.944098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.943952    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gh4\" (UniqueName: \"kubernetes.io/projected/293eae7c-c172-4580-b4d4-5933e97ed0fc-kube-api-access-h4gh4\") pod \"maas-api-revoke-sub-3bab26dc-5dxkp\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:31.981044 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:31.981010    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:32.011380 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:32.011337    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:32.338953 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:32.338920    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"]
Oct 29 23:34:32.340165 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:32.340134    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293eae7c_c172_4580_b4d4_5933e97ed0fc.slice/crio-e5fe69734e3417c5cd3cf98e96d3ae52e2ab66ca598f99a53725453ba757085d WatchSource:0}: Error finding container e5fe69734e3417c5cd3cf98e96d3ae52e2ab66ca598f99a53725453ba757085d: Status 404 returned error can't find the container with id e5fe69734e3417c5cd3cf98e96d3ae52e2ab66ca598f99a53725453ba757085d
Oct 29 23:34:32.418389 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:32.418354    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"]
Oct 29 23:34:32.420457 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:32.420420    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b817b21_4bf2_415a_928b_cb2f20a5b995.slice/crio-f336e3077a1714879961368598da433cc9776d97b1aa30c9a345273955c61064 WatchSource:0}: Error finding container f336e3077a1714879961368598da433cc9776d97b1aa30c9a345273955c61064: Status 404 returned error can't find the container with id f336e3077a1714879961368598da433cc9776d97b1aa30c9a345273955c61064
Oct 29 23:34:32.721703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:32.721600    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:32.722167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:32.722141    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:32.723520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:32.723457    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:34:33.041955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.041853    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt" event={"ID":"8b817b21-4bf2-415a-928b-cb2f20a5b995","Type":"ContainerStarted","Data":"62355e36bb40fdbd61c93f5f88252300696aeaf2d6e577759b6a28164cc14998"}
Oct 29 23:34:33.041955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.041945    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt" event={"ID":"8b817b21-4bf2-415a-928b-cb2f20a5b995","Type":"ContainerStarted","Data":"f336e3077a1714879961368598da433cc9776d97b1aa30c9a345273955c61064"}
Oct 29 23:34:33.043731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.043706    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp" event={"ID":"293eae7c-c172-4580-b4d4-5933e97ed0fc","Type":"ContainerStarted","Data":"0247f87c2169e4fa9f49e6d0c0322b2710a808a93067981fbcef502358aa4f7f"}
Oct 29 23:34:33.043834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.043753    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp" event={"ID":"293eae7c-c172-4580-b4d4-5933e97ed0fc","Type":"ContainerStarted","Data":"e5fe69734e3417c5cd3cf98e96d3ae52e2ab66ca598f99a53725453ba757085d"}
Oct 29 23:34:33.063841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.063803    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main" containerID="cri-o://bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b" gracePeriod=45
Oct 29 23:34:33.099096 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.099042    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt" podStartSLOduration=2.0990241689999998 podStartE2EDuration="2.099024169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:33.097078313 +0000 UTC m=+1251.201338699" watchObservedRunningTime="2026-10-29 23:34:33.099024169 +0000 UTC m=+1251.203284546"
Oct 29 23:34:33.145866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.145818    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp" podStartSLOduration=2.145805544 podStartE2EDuration="2.145805544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:33.145589109 +0000 UTC m=+1251.249849495" watchObservedRunningTime="2026-10-29 23:34:33.145805544 +0000 UTC m=+1251.250065929"
Oct 29 23:34:33.353792 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.353765    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:34:33.443497 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.443466    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c580d5bd-73c2-4010-8087-61d8100e8df7-kube-api-access-tmhwh\" (UniqueName: \"kubernetes.io/projected/c580d5bd-73c2-4010-8087-61d8100e8df7-kube-api-access-tmhwh\") pod \"c580d5bd-73c2-4010-8087-61d8100e8df7\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") "
Oct 29 23:34:33.443636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.443514    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-home\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-home\") pod \"c580d5bd-73c2-4010-8087-61d8100e8df7\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") "
Oct 29 23:34:33.443699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.443649    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c580d5bd-73c2-4010-8087-61d8100e8df7-tls-certs\" (UniqueName: \"kubernetes.io/secret/c580d5bd-73c2-4010-8087-61d8100e8df7-tls-certs\") pod \"c580d5bd-73c2-4010-8087-61d8100e8df7\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") "
Oct 29 23:34:33.443775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.443755    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-home" pod "c580d5bd-73c2-4010-8087-61d8100e8df7" (UID: "c580d5bd-73c2-4010-8087-61d8100e8df7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:33.443832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.443785    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-model-cache\") pod \"c580d5bd-73c2-4010-8087-61d8100e8df7\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") "
Oct 29 23:34:33.443979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.443959    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-model-cache" pod "c580d5bd-73c2-4010-8087-61d8100e8df7" (UID: "c580d5bd-73c2-4010-8087-61d8100e8df7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:33.444048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.443977    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-tmp-dir\") pod \"c580d5bd-73c2-4010-8087-61d8100e8df7\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") "
Oct 29 23:34:33.444048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.444009    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-dshm\") pod \"c580d5bd-73c2-4010-8087-61d8100e8df7\" (UID: \"c580d5bd-73c2-4010-8087-61d8100e8df7\") "
Oct 29 23:34:33.444244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.444225    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-tmp-dir" pod "c580d5bd-73c2-4010-8087-61d8100e8df7" (UID: "c580d5bd-73c2-4010-8087-61d8100e8df7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:33.444619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.444592    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:33.444725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.444621    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:33.444725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.444634    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:33.445918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.445858    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c580d5bd-73c2-4010-8087-61d8100e8df7-tls-certs" pod "c580d5bd-73c2-4010-8087-61d8100e8df7" (UID: "c580d5bd-73c2-4010-8087-61d8100e8df7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:33.446041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.445927    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c580d5bd-73c2-4010-8087-61d8100e8df7-kube-api-access-tmhwh" pod "c580d5bd-73c2-4010-8087-61d8100e8df7" (UID: "c580d5bd-73c2-4010-8087-61d8100e8df7"). InnerVolumeSpecName "kube-api-access-tmhwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:33.446132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.446112    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-dshm" pod "c580d5bd-73c2-4010-8087-61d8100e8df7" (UID: "c580d5bd-73c2-4010-8087-61d8100e8df7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:34:33.546493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.546458    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c580d5bd-73c2-4010-8087-61d8100e8df7-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:33.546668 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.546497    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmhwh\" (UniqueName: \"kubernetes.io/projected/c580d5bd-73c2-4010-8087-61d8100e8df7-kube-api-access-tmhwh\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:33.546668 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:33.546514    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c580d5bd-73c2-4010-8087-61d8100e8df7-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:34.053019 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.052982    6203 generic.go:356] "Generic (PLEG): container finished" podID="293eae7c-c172-4580-b4d4-5933e97ed0fc" containerID="0247f87c2169e4fa9f49e6d0c0322b2710a808a93067981fbcef502358aa4f7f" exitCode=0
Oct 29 23:34:34.053400 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.053071    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp" event={"ID":"293eae7c-c172-4580-b4d4-5933e97ed0fc","Type":"ContainerDied","Data":"0247f87c2169e4fa9f49e6d0c0322b2710a808a93067981fbcef502358aa4f7f"}
Oct 29 23:34:34.054562 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.054535    6203 generic.go:356] "Generic (PLEG): container finished" podID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerID="bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b" exitCode=0
Oct 29 23:34:34.054693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.054583    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"
Oct 29 23:34:34.054693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.054614    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" event={"ID":"c580d5bd-73c2-4010-8087-61d8100e8df7","Type":"ContainerDied","Data":"bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b"}
Oct 29 23:34:34.054693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.054647    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc" event={"ID":"c580d5bd-73c2-4010-8087-61d8100e8df7","Type":"ContainerDied","Data":"4e3c6817dc29880111b49a558725ccfec929c21ea25fead391e438fc0e393a37"}
Oct 29 23:34:34.054693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.054664    6203 scope.go:122] "RemoveContainer" containerID="bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b"
Oct 29 23:34:34.056419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.056393    6203 generic.go:356] "Generic (PLEG): container finished" podID="8b817b21-4bf2-415a-928b-cb2f20a5b995" containerID="62355e36bb40fdbd61c93f5f88252300696aeaf2d6e577759b6a28164cc14998" exitCode=0
Oct 29 23:34:34.056540 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.056470    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt" event={"ID":"8b817b21-4bf2-415a-928b-cb2f20a5b995","Type":"ContainerDied","Data":"62355e36bb40fdbd61c93f5f88252300696aeaf2d6e577759b6a28164cc14998"}
Oct 29 23:34:34.065685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.065485    6203 scope.go:122] "RemoveContainer" containerID="bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b"
Oct 29 23:34:34.065813 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:34:34.065793    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b\": container with ID starting with bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b not found: ID does not exist" containerID="bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b"
Oct 29 23:34:34.065863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.065822    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b"} err="failed to get container status \"bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b\": rpc error: code = NotFound desc = could not find container \"bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b\": container with ID starting with bd73d243e430d2a1247196bb0516f71f2844970f3e4e2155ec1558407fbe568b not found: ID does not exist"
Oct 29 23:34:34.159233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.159195    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"]
Oct 29 23:34:34.163533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.163505    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2a53c9/premium-w1-2a53c9-kserve-756f558846-8vrwc"]
Oct 29 23:34:34.238790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.238713    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"]
Oct 29 23:34:34.239300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.239285    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main"
Oct 29 23:34:34.239348 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.239302    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main"
Oct 29 23:34:34.239436 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.239428    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" containerName="main"
Oct 29 23:34:34.244210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.244187    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.256461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.256328    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.256461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.256457    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fhs\" (UniqueName: \"kubernetes.io/projected/2156d51e-1686-4d52-9c06-3aee9f8e15f6-kube-api-access-v5fhs\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.256671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.256589    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.269209 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.264506    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"]
Oct 29 23:34:34.358149 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.358115    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.358326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.358165    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fhs\" (UniqueName: \"kubernetes.io/projected/2156d51e-1686-4d52-9c06-3aee9f8e15f6-kube-api-access-v5fhs\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.358326 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.358237    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.358729 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.358707    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.360617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.360597    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.384117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.384087    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fhs\" (UniqueName: \"kubernetes.io/projected/2156d51e-1686-4d52-9c06-3aee9f8e15f6-kube-api-access-v5fhs\") pod \"maas-api-revoke-sub-dde306b2-4w6lt\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.558079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.557997    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:34.861380 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:34.861350    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2156d51e_1686_4d52_9c06_3aee9f8e15f6.slice/crio-438eae0b27eb97f2dc3a411e81e343c97ec4fb3ff6520f497e61e1b7c5aadaf7 WatchSource:0}: Error finding container 438eae0b27eb97f2dc3a411e81e343c97ec4fb3ff6520f497e61e1b7c5aadaf7: Status 404 returned error can't find the container with id 438eae0b27eb97f2dc3a411e81e343c97ec4fb3ff6520f497e61e1b7c5aadaf7
Oct 29 23:34:34.862221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:34.862195    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"]
Oct 29 23:34:35.067227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.067191    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt" event={"ID":"2156d51e-1686-4d52-9c06-3aee9f8e15f6","Type":"ContainerStarted","Data":"438eae0b27eb97f2dc3a411e81e343c97ec4fb3ff6520f497e61e1b7c5aadaf7"}
Oct 29 23:34:35.265381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.265355    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:35.268670 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.268647    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:35.371785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.371751    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293eae7c-c172-4580-b4d4-5933e97ed0fc-kube-api-access-h4gh4\" (UniqueName: \"kubernetes.io/projected/293eae7c-c172-4580-b4d4-5933e97ed0fc-kube-api-access-h4gh4\") pod \"293eae7c-c172-4580-b4d4-5933e97ed0fc\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") "
Oct 29 23:34:35.371987 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.371937    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-xks-ca\") pod \"8b817b21-4bf2-415a-928b-cb2f20a5b995\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") "
Oct 29 23:34:35.371987 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.371979    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-openshift-ca\") pod \"8b817b21-4bf2-415a-928b-cb2f20a5b995\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") "
Oct 29 23:34:35.372120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.372007    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-xks-ca\") pod \"293eae7c-c172-4580-b4d4-5933e97ed0fc\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") "
Oct 29 23:34:35.372179 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.372152    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-openshift-ca\") pod \"293eae7c-c172-4580-b4d4-5933e97ed0fc\" (UID: \"293eae7c-c172-4580-b4d4-5933e97ed0fc\") "
Oct 29 23:34:35.372267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.372244    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b817b21-4bf2-415a-928b-cb2f20a5b995-kube-api-access-tfkkf\" (UniqueName: \"kubernetes.io/projected/8b817b21-4bf2-415a-928b-cb2f20a5b995-kube-api-access-tfkkf\") pod \"8b817b21-4bf2-415a-928b-cb2f20a5b995\" (UID: \"8b817b21-4bf2-415a-928b-cb2f20a5b995\") "
Oct 29 23:34:35.372574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.372547    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-openshift-ca" pod "8b817b21-4bf2-415a-928b-cb2f20a5b995" (UID: "8b817b21-4bf2-415a-928b-cb2f20a5b995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:35.372920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.372858    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:35.373028 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.372969    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-openshift-ca" pod "293eae7c-c172-4580-b4d4-5933e97ed0fc" (UID: "293eae7c-c172-4580-b4d4-5933e97ed0fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:35.375072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.375013    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293eae7c-c172-4580-b4d4-5933e97ed0fc-kube-api-access-h4gh4" pod "293eae7c-c172-4580-b4d4-5933e97ed0fc" (UID: "293eae7c-c172-4580-b4d4-5933e97ed0fc"). InnerVolumeSpecName "kube-api-access-h4gh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:35.375174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.375114    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-xks-ca" pod "8b817b21-4bf2-415a-928b-cb2f20a5b995" (UID: "8b817b21-4bf2-415a-928b-cb2f20a5b995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:35.375230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.375185    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b817b21-4bf2-415a-928b-cb2f20a5b995-kube-api-access-tfkkf" pod "8b817b21-4bf2-415a-928b-cb2f20a5b995" (UID: "8b817b21-4bf2-415a-928b-cb2f20a5b995"). InnerVolumeSpecName "kube-api-access-tfkkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:35.375941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.375919    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-xks-ca" pod "293eae7c-c172-4580-b4d4-5933e97ed0fc" (UID: "293eae7c-c172-4580-b4d4-5933e97ed0fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:35.474665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.474630    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b817b21-4bf2-415a-928b-cb2f20a5b995-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:35.474665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.474662    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:35.474665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.474671    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/293eae7c-c172-4580-b4d4-5933e97ed0fc-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:35.474937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.474680    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfkkf\" (UniqueName: \"kubernetes.io/projected/8b817b21-4bf2-415a-928b-cb2f20a5b995-kube-api-access-tfkkf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:35.474937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:35.474690    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4gh4\" (UniqueName: \"kubernetes.io/projected/293eae7c-c172-4580-b4d4-5933e97ed0fc-kube-api-access-h4gh4\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:36.019053 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.018959    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c580d5bd-73c2-4010-8087-61d8100e8df7" path="/var/lib/kubelet/pods/c580d5bd-73c2-4010-8087-61d8100e8df7/volumes"
Oct 29 23:34:36.078338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.078291    6203 generic.go:356] "Generic (PLEG): container finished" podID="2156d51e-1686-4d52-9c06-3aee9f8e15f6" containerID="45eef163b09bd1221ec5c3e6a55c120f20e747bd0718861a1e9a4050ae1641db" exitCode=0
Oct 29 23:34:36.078815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.078494    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt" event={"ID":"2156d51e-1686-4d52-9c06-3aee9f8e15f6","Type":"ContainerDied","Data":"45eef163b09bd1221ec5c3e6a55c120f20e747bd0718861a1e9a4050ae1641db"}
Oct 29 23:34:36.080251 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.080227    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"
Oct 29 23:34:36.080395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.080249    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp" event={"ID":"293eae7c-c172-4580-b4d4-5933e97ed0fc","Type":"ContainerDied","Data":"e5fe69734e3417c5cd3cf98e96d3ae52e2ab66ca598f99a53725453ba757085d"}
Oct 29 23:34:36.080395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.080293    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5fe69734e3417c5cd3cf98e96d3ae52e2ab66ca598f99a53725453ba757085d"
Oct 29 23:34:36.081808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.081782    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"
Oct 29 23:34:36.081808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.081798    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt" event={"ID":"8b817b21-4bf2-415a-928b-cb2f20a5b995","Type":"ContainerDied","Data":"f336e3077a1714879961368598da433cc9776d97b1aa30c9a345273955c61064"}
Oct 29 23:34:36.081959 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:36.081833    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f336e3077a1714879961368598da433cc9776d97b1aa30c9a345273955c61064"
Oct 29 23:34:37.711419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.711039    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:37.816913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.816846    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-xks-ca\") pod \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") "
Oct 29 23:34:37.817112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.817056    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-openshift-ca\") pod \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") "
Oct 29 23:34:37.817238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.817214    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2156d51e-1686-4d52-9c06-3aee9f8e15f6-kube-api-access-v5fhs\" (UniqueName: \"kubernetes.io/projected/2156d51e-1686-4d52-9c06-3aee9f8e15f6-kube-api-access-v5fhs\") pod \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\" (UID: \"2156d51e-1686-4d52-9c06-3aee9f8e15f6\") "
Oct 29 23:34:37.818837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.818803    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-openshift-ca" pod "2156d51e-1686-4d52-9c06-3aee9f8e15f6" (UID: "2156d51e-1686-4d52-9c06-3aee9f8e15f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:37.821100 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.821071    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-xks-ca" pod "2156d51e-1686-4d52-9c06-3aee9f8e15f6" (UID: "2156d51e-1686-4d52-9c06-3aee9f8e15f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:37.874852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.874604    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2156d51e-1686-4d52-9c06-3aee9f8e15f6-kube-api-access-v5fhs" pod "2156d51e-1686-4d52-9c06-3aee9f8e15f6" (UID: "2156d51e-1686-4d52-9c06-3aee9f8e15f6"). InnerVolumeSpecName "kube-api-access-v5fhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:37.920379 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.920342    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5fhs\" (UniqueName: \"kubernetes.io/projected/2156d51e-1686-4d52-9c06-3aee9f8e15f6-kube-api-access-v5fhs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:37.920379 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.920378    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:37.920581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:37.920391    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2156d51e-1686-4d52-9c06-3aee9f8e15f6-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:38.126640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:38.126564    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"
Oct 29 23:34:38.126789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:38.126563    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt" event={"ID":"2156d51e-1686-4d52-9c06-3aee9f8e15f6","Type":"ContainerDied","Data":"438eae0b27eb97f2dc3a411e81e343c97ec4fb3ff6520f497e61e1b7c5aadaf7"}
Oct 29 23:34:38.126789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:38.126685    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="438eae0b27eb97f2dc3a411e81e343c97ec4fb3ff6520f497e61e1b7c5aadaf7"
Oct 29 23:34:39.050611 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:39.050578    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"]
Oct 29 23:34:39.064904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:39.064846    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4491bb74-4w4r4"]
Oct 29 23:34:39.923850 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:34:39.923817    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:34:40.019069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:40.019034    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec21f45-e3a9-4aff-b528-d5babb961453" path="/var/lib/kubelet/pods/7ec21f45-e3a9-4aff-b528-d5babb961453/volumes"
Oct 29 23:34:41.033119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:41.033084    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"]
Oct 29 23:34:41.047064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:41.047035    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c021d202-lzz7x"]
Oct 29 23:34:42.019635 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:42.019600    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed424d6f-3464-4853-b1f2-ec7eef095c5b" path="/var/lib/kubelet/pods/ed424d6f-3464-4853-b1f2-ec7eef095c5b/volumes"
Oct 29 23:34:43.043771 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:43.043740    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"]
Oct 29 23:34:43.058260 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:43.058227    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7199a7-5rvmh"]
Oct 29 23:34:44.018922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.018853    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf87280a-7926-4446-85f5-294bf2a519ef" path="/var/lib/kubelet/pods/cf87280a-7926-4446-85f5-294bf2a519ef/volumes"
Oct 29 23:34:44.242521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.242482    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"]
Oct 29 23:34:44.243070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243056    6203 cpu_manager.go:414] "containerMap: removing container" podUID="293eae7c-c172-4580-b4d4-5933e97ed0fc" containerName="revoke-keys"
Oct 29 23:34:44.243119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243071    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293eae7c-c172-4580-b4d4-5933e97ed0fc" containerName="revoke-keys"
Oct 29 23:34:44.243119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243102    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2156d51e-1686-4d52-9c06-3aee9f8e15f6" containerName="revoke-keys"
Oct 29 23:34:44.243119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243108    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2156d51e-1686-4d52-9c06-3aee9f8e15f6" containerName="revoke-keys"
Oct 29 23:34:44.243119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243118    6203 cpu_manager.go:414] "containerMap: removing container" podUID="8b817b21-4bf2-415a-928b-cb2f20a5b995" containerName="revoke-keys"
Oct 29 23:34:44.243240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243124    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b817b21-4bf2-415a-928b-cb2f20a5b995" containerName="revoke-keys"
Oct 29 23:34:44.243240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243210    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2156d51e-1686-4d52-9c06-3aee9f8e15f6" containerName="revoke-keys"
Oct 29 23:34:44.243240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243219    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b817b21-4bf2-415a-928b-cb2f20a5b995" containerName="revoke-keys"
Oct 29 23:34:44.243240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.243228    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="293eae7c-c172-4580-b4d4-5933e97ed0fc" containerName="revoke-keys"
Oct 29 23:34:44.248712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.248687    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.270290 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.270221    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"]
Oct 29 23:34:44.279677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.279636    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:34:44.279677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.279636    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:34:44.279677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.279648    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:34:44.281709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.281683    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:34:44.312103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.312062    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzzkf\" (UniqueName: \"kubernetes.io/projected/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-kube-api-access-bzzkf\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.312336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.312171    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.312336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.312252    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.414330 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.414282    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzzkf\" (UniqueName: \"kubernetes.io/projected/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-kube-api-access-bzzkf\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.414539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.414422    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.414539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.414511    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.416734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.416696    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.424377 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.424344    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.444830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.444788    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzzkf\" (UniqueName: \"kubernetes.io/projected/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-kube-api-access-bzzkf\") pod \"maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.564945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.564843    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:44.876641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:44.876607    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"]
Oct 29 23:34:44.876991 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:44.876968    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda854c1eb_67ea_4b7c_9dd2_9d02595b53f6.slice/crio-d112efe9ab0279b32c9c7532f505c63e42c28c464b28d2c32e33d266d846f464 WatchSource:0}: Error finding container d112efe9ab0279b32c9c7532f505c63e42c28c464b28d2c32e33d266d846f464: Status 404 returned error can't find the container with id d112efe9ab0279b32c9c7532f505c63e42c28c464b28d2c32e33d266d846f464
Oct 29 23:34:45.032940 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:45.032902    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"]
Oct 29 23:34:45.036842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:45.036810    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a03f81e-r6f9x"]
Oct 29 23:34:45.182913 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:45.182853    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh" event={"ID":"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6","Type":"ContainerStarted","Data":"b65b9e6a244f81871f48dee017cc75423e5b389c6fddf9d34e1bc382aa750d89"}
Oct 29 23:34:45.183103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:45.182930    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh" event={"ID":"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6","Type":"ContainerStarted","Data":"d112efe9ab0279b32c9c7532f505c63e42c28c464b28d2c32e33d266d846f464"}
Oct 29 23:34:45.229783 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:45.229730    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh" podStartSLOduration=1.229715058 podStartE2EDuration="1.229715058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:45.227361704 +0000 UTC m=+1263.331622091" watchObservedRunningTime="2026-10-29 23:34:45.229715058 +0000 UTC m=+1263.333975461"
Oct 29 23:34:46.021700 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:46.021670    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9c42cf-9e27-4021-8682-3ca0e5d916f4" path="/var/lib/kubelet/pods/3f9c42cf-9e27-4021-8682-3ca0e5d916f4/volumes"
Oct 29 23:34:46.192541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:46.192504    6203 generic.go:356] "Generic (PLEG): container finished" podID="a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" containerID="b65b9e6a244f81871f48dee017cc75423e5b389c6fddf9d34e1bc382aa750d89" exitCode=0
Oct 29 23:34:46.192720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:46.192590    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh" event={"ID":"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6","Type":"ContainerDied","Data":"b65b9e6a244f81871f48dee017cc75423e5b389c6fddf9d34e1bc382aa750d89"}
Oct 29 23:34:46.449120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:46.449083    6203 scope.go:122] "RemoveContainer" containerID="f8c174fb0a1ec584c1a6dc060330e42ebec8b9ccb87abee027dbeb05e333e8ea"
Oct 29 23:34:47.376951 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.376845    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:47.479997 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.479903    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-kube-api-access-bzzkf\" (UniqueName: \"kubernetes.io/projected/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-kube-api-access-bzzkf\") pod \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") "
Oct 29 23:34:47.480175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.480066    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-openshift-ca\") pod \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") "
Oct 29 23:34:47.480175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.480150    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-xks-ca\") pod \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\" (UID: \"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6\") "
Oct 29 23:34:47.481092 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.481061    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-openshift-ca" pod "a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" (UID: "a854c1eb-67ea-4b7c-9dd2-9d02595b53f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:47.487242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.487179    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-kube-api-access-bzzkf" pod "a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" (UID: "a854c1eb-67ea-4b7c-9dd2-9d02595b53f6"). InnerVolumeSpecName "kube-api-access-bzzkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:47.487242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.487195    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-xks-ca" pod "a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" (UID: "a854c1eb-67ea-4b7c-9dd2-9d02595b53f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:47.582096 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.582056    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:47.582096 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.582089    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:47.582096 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.582098    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzzkf\" (UniqueName: \"kubernetes.io/projected/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6-kube-api-access-bzzkf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:47.605125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.605093    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"]
Oct 29 23:34:47.608931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:47.608898    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-2a53c9-t9f2b"]
Oct 29 23:34:48.019125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:48.019094    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619a55e9-8751-480c-a854-5b8c5421f6e5" path="/var/lib/kubelet/pods/619a55e9-8751-480c-a854-5b8c5421f6e5/volumes"
Oct 29 23:34:48.208407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:48.208377    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"
Oct 29 23:34:48.208407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:48.208399    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh" event={"ID":"a854c1eb-67ea-4b7c-9dd2-9d02595b53f6","Type":"ContainerDied","Data":"d112efe9ab0279b32c9c7532f505c63e42c28c464b28d2c32e33d266d846f464"}
Oct 29 23:34:48.208617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:48.208443    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d112efe9ab0279b32c9c7532f505c63e42c28c464b28d2c32e33d266d846f464"
Oct 29 23:34:49.045378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.045344    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"]
Oct 29 23:34:49.062596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.062563    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"]
Oct 29 23:34:49.384757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.384681    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"]
Oct 29 23:34:49.385313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.385287    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" containerName="revoke-keys"
Oct 29 23:34:49.385313 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.385315    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" containerName="revoke-keys"
Oct 29 23:34:49.385432 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.385422    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" containerName="revoke-keys"
Oct 29 23:34:49.389320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.389304    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.409236 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.409208    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"]
Oct 29 23:34:49.415463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.415437    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:34:49.415688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.415437    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:34:49.415688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.415506    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:34:49.415864 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.415595    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:34:49.511170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.511133    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.511364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.511216    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.511447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.511395    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtvd\" (UniqueName: \"kubernetes.io/projected/eb491355-dec4-44b0-8c70-cbcadade530f-kube-api-access-4vtvd\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.613322 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.613269    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.613551 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.613449    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtvd\" (UniqueName: \"kubernetes.io/projected/eb491355-dec4-44b0-8c70-cbcadade530f-kube-api-access-4vtvd\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.614304 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.614228    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.614828 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.614785    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.630683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.630587    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.650409 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.650321    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtvd\" (UniqueName: \"kubernetes.io/projected/eb491355-dec4-44b0-8c70-cbcadade530f-kube-api-access-4vtvd\") pod \"maas-api-revoke-sub-54e704d6-zfv7r\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:49.704050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:49.704013    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:50.010959 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:50.010923    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb491355_dec4_44b0_8c70_cbcadade530f.slice/crio-a4e1733084493bf81d28c39b1bfc025ef9edb329f4c2f7d7a549817f31aece52 WatchSource:0}: Error finding container a4e1733084493bf81d28c39b1bfc025ef9edb329f4c2f7d7a549817f31aece52: Status 404 returned error can't find the container with id a4e1733084493bf81d28c39b1bfc025ef9edb329f4c2f7d7a549817f31aece52
Oct 29 23:34:50.019733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:50.019710    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"]
Oct 29 23:34:50.049922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:50.049896    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"]
Oct 29 23:34:50.055709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:50.055684    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64b99370-bdmj5"]
Oct 29 23:34:50.225113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:50.225080    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r" event={"ID":"eb491355-dec4-44b0-8c70-cbcadade530f","Type":"ContainerStarted","Data":"a4e1733084493bf81d28c39b1bfc025ef9edb329f4c2f7d7a549817f31aece52"}
Oct 29 23:34:51.031974 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:51.031941    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"]
Oct 29 23:34:51.045932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:51.045900    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9dc6892-59mzm"]
Oct 29 23:34:51.242636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:51.242532    6203 generic.go:356] "Generic (PLEG): container finished" podID="eb491355-dec4-44b0-8c70-cbcadade530f" containerID="fc032eee016705e5a9bde975f243d79a3c7f1931ae98641e86efc47e6017f3bd" exitCode=0
Oct 29 23:34:51.242636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:51.242621    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r" event={"ID":"eb491355-dec4-44b0-8c70-cbcadade530f","Type":"ContainerDied","Data":"fc032eee016705e5a9bde975f243d79a3c7f1931ae98641e86efc47e6017f3bd"}
Oct 29 23:34:52.019274 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.019231    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1c969c-0ca7-4083-b33d-fd81bdc20aad" path="/var/lib/kubelet/pods/8c1c969c-0ca7-4083-b33d-fd81bdc20aad/volumes"
Oct 29 23:34:52.020007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.019979    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7029084-02e0-48d9-a3bc-4f67d12c4297" path="/var/lib/kubelet/pods/f7029084-02e0-48d9-a3bc-4f67d12c4297/volumes"
Oct 29 23:34:52.491935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.491907    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:52.557396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.556318    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb491355-dec4-44b0-8c70-cbcadade530f-kube-api-access-4vtvd\" (UniqueName: \"kubernetes.io/projected/eb491355-dec4-44b0-8c70-cbcadade530f-kube-api-access-4vtvd\") pod \"eb491355-dec4-44b0-8c70-cbcadade530f\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") "
Oct 29 23:34:52.557396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.556527    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-openshift-ca\") pod \"eb491355-dec4-44b0-8c70-cbcadade530f\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") "
Oct 29 23:34:52.557396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.556568    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-xks-ca\") pod \"eb491355-dec4-44b0-8c70-cbcadade530f\" (UID: \"eb491355-dec4-44b0-8c70-cbcadade530f\") "
Oct 29 23:34:52.558174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.558130    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-openshift-ca" pod "eb491355-dec4-44b0-8c70-cbcadade530f" (UID: "eb491355-dec4-44b0-8c70-cbcadade530f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:52.570305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.570264    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-xks-ca" pod "eb491355-dec4-44b0-8c70-cbcadade530f" (UID: "eb491355-dec4-44b0-8c70-cbcadade530f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:52.572167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.572096    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb491355-dec4-44b0-8c70-cbcadade530f-kube-api-access-4vtvd" pod "eb491355-dec4-44b0-8c70-cbcadade530f" (UID: "eb491355-dec4-44b0-8c70-cbcadade530f"). InnerVolumeSpecName "kube-api-access-4vtvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:52.679903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.660902    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:52.679903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.660947    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb491355-dec4-44b0-8c70-cbcadade530f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:52.679903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:52.660963    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtvd\" (UniqueName: \"kubernetes.io/projected/eb491355-dec4-44b0-8c70-cbcadade530f-kube-api-access-4vtvd\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:53.058904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.058099    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"]
Oct 29 23:34:53.065363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.063965    6203 cpu_manager.go:414] "containerMap: removing container" podUID="eb491355-dec4-44b0-8c70-cbcadade530f" containerName="revoke-keys"
Oct 29 23:34:53.065363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.063991    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb491355-dec4-44b0-8c70-cbcadade530f" containerName="revoke-keys"
Oct 29 23:34:53.065363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.064206    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb491355-dec4-44b0-8c70-cbcadade530f" containerName="revoke-keys"
Oct 29 23:34:53.120305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.120182    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"]
Oct 29 23:34:53.120497 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.120344    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.182863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.182718    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.186275 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.183046    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.186275 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.183146    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/7336aa02-bb14-41d7-aa2d-9e69b19812a9-kube-api-access-tfmmf\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.261108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.261081    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"
Oct 29 23:34:53.261108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.261097    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r" event={"ID":"eb491355-dec4-44b0-8c70-cbcadade530f","Type":"ContainerDied","Data":"a4e1733084493bf81d28c39b1bfc025ef9edb329f4c2f7d7a549817f31aece52"}
Oct 29 23:34:53.261319 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.261152    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e1733084493bf81d28c39b1bfc025ef9edb329f4c2f7d7a549817f31aece52"
Oct 29 23:34:53.284634 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.284601    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.284791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.284729    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.284841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.284788    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/7336aa02-bb14-41d7-aa2d-9e69b19812a9-kube-api-access-tfmmf\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.285519 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.285481    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.287161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.287136    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.310493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.310430    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/7336aa02-bb14-41d7-aa2d-9e69b19812a9-kube-api-access-tfmmf\") pod \"maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.492647 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.492619    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:53.804952 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:53.804914    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7336aa02_bb14_41d7_aa2d_9e69b19812a9.slice/crio-47dfa7dabc7f5953ad36aa475c7ca69347544d4cb34e367a4cbdd8143a9dc35b WatchSource:0}: Error finding container 47dfa7dabc7f5953ad36aa475c7ca69347544d4cb34e367a4cbdd8143a9dc35b: Status 404 returned error can't find the container with id 47dfa7dabc7f5953ad36aa475c7ca69347544d4cb34e367a4cbdd8143a9dc35b
Oct 29 23:34:53.806194 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:53.806169    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"]
Oct 29 23:34:54.032383 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:54.032285    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"]
Oct 29 23:34:54.036546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:54.036518    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bce274a6-rt7f9"]
Oct 29 23:34:54.270833 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:54.270799    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw" event={"ID":"7336aa02-bb14-41d7-aa2d-9e69b19812a9","Type":"ContainerStarted","Data":"7b750667a8864dcca25fd7e2fa27663b2d8d441ad6bbbb8fbfcda33d10ce2049"}
Oct 29 23:34:54.271062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:54.270857    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw" event={"ID":"7336aa02-bb14-41d7-aa2d-9e69b19812a9","Type":"ContainerStarted","Data":"47dfa7dabc7f5953ad36aa475c7ca69347544d4cb34e367a4cbdd8143a9dc35b"}
Oct 29 23:34:54.766053 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:54.766001    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw" podStartSLOduration=1.765982851 podStartE2EDuration="1.765982851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:54.325697479 +0000 UTC m=+1272.429957867" watchObservedRunningTime="2026-10-29 23:34:54.765982851 +0000 UTC m=+1272.870243237"
Oct 29 23:34:54.767483 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:54.767464    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"]
Oct 29 23:34:54.797526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:54.797492    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"]
Oct 29 23:34:55.279751 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.279717    6203 generic.go:356] "Generic (PLEG): container finished" podID="7336aa02-bb14-41d7-aa2d-9e69b19812a9" containerID="7b750667a8864dcca25fd7e2fa27663b2d8d441ad6bbbb8fbfcda33d10ce2049" exitCode=0
Oct 29 23:34:55.279946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.279805    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw" event={"ID":"7336aa02-bb14-41d7-aa2d-9e69b19812a9","Type":"ContainerDied","Data":"7b750667a8864dcca25fd7e2fa27663b2d8d441ad6bbbb8fbfcda33d10ce2049"}
Oct 29 23:34:55.756496 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.756462    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"]
Oct 29 23:34:55.756735 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.756710    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy" containerID="cri-o://6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135" gracePeriod=30
Oct 29 23:34:55.765232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.765198    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"]
Oct 29 23:34:55.765458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.765432    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy" containerID="cri-o://cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463" gracePeriod=30
Oct 29 23:34:55.769378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.769356    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"]
Oct 29 23:34:55.769706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:55.769519    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy" containerID="cri-o://5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7" gracePeriod=30
Oct 29 23:34:56.018851 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.018764    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd" path="/var/lib/kubelet/pods/5f6e1fab-faa1-44d6-bfaa-2e0ac80317bd/volumes"
Oct 29 23:34:56.042404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.042372    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"]
Oct 29 23:34:56.046484 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.046458    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fb3083-fmktt"]
Oct 29 23:34:56.482317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.482290    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:56.565094 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.564994    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7336aa02-bb14-41d7-aa2d-9e69b19812a9-kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/7336aa02-bb14-41d7-aa2d-9e69b19812a9-kube-api-access-tfmmf\") pod \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") "
Oct 29 23:34:56.565272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.565162    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-openshift-ca\") pod \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") "
Oct 29 23:34:56.565272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.565212    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-xks-ca\") pod \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\" (UID: \"7336aa02-bb14-41d7-aa2d-9e69b19812a9\") "
Oct 29 23:34:56.567353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.567309    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-openshift-ca" pod "7336aa02-bb14-41d7-aa2d-9e69b19812a9" (UID: "7336aa02-bb14-41d7-aa2d-9e69b19812a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:34:56.568266 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.568241    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-xks-ca" pod "7336aa02-bb14-41d7-aa2d-9e69b19812a9" (UID: "7336aa02-bb14-41d7-aa2d-9e69b19812a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:34:56.569758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.569729    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7336aa02-bb14-41d7-aa2d-9e69b19812a9-kube-api-access-tfmmf" pod "7336aa02-bb14-41d7-aa2d-9e69b19812a9" (UID: "7336aa02-bb14-41d7-aa2d-9e69b19812a9"). InnerVolumeSpecName "kube-api-access-tfmmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:34:56.666573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.666523    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:56.666573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.666570    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7336aa02-bb14-41d7-aa2d-9e69b19812a9-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:56.666573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:56.666584    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfmmf\" (UniqueName: \"kubernetes.io/projected/7336aa02-bb14-41d7-aa2d-9e69b19812a9-kube-api-access-tfmmf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:34:57.296505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:57.296474    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw" event={"ID":"7336aa02-bb14-41d7-aa2d-9e69b19812a9","Type":"ContainerDied","Data":"47dfa7dabc7f5953ad36aa475c7ca69347544d4cb34e367a4cbdd8143a9dc35b"}
Oct 29 23:34:57.296505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:57.296492    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"
Oct 29 23:34:57.296948 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:57.296527    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47dfa7dabc7f5953ad36aa475c7ca69347544d4cb34e367a4cbdd8143a9dc35b"
Oct 29 23:34:57.815481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:57.815445    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"]
Oct 29 23:34:57.825507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:57.825478    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"]
Oct 29 23:34:58.032051 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.031997    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e27aa1-da0e-412d-9f54-c0b3289fc469" path="/var/lib/kubelet/pods/e0e27aa1-da0e-412d-9f54-c0b3289fc469/volumes"
Oct 29 23:34:58.034512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.034486    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"]
Oct 29 23:34:58.038249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.038222    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d891a57f-mwn8g"]
Oct 29 23:34:58.161419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.161341    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"]
Oct 29 23:34:58.161937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.161923    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7336aa02-bb14-41d7-aa2d-9e69b19812a9" containerName="revoke-keys"
Oct 29 23:34:58.161978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.161938    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7336aa02-bb14-41d7-aa2d-9e69b19812a9" containerName="revoke-keys"
Oct 29 23:34:58.162033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.162027    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7336aa02-bb14-41d7-aa2d-9e69b19812a9" containerName="revoke-keys"
Oct 29 23:34:58.166996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.166977    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.184014 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.183980    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.184238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.184116    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.184238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.184146    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjkx\" (UniqueName: \"kubernetes.io/projected/d4d50f3d-7e8a-4a14-9864-3d9525668f82-kube-api-access-hzjkx\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.187703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.187665    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:34:58.187857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.187721    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:34:58.188018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.187988    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:34:58.188143 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.188111    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:34:58.188764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.188749    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"]
Oct 29 23:34:58.194012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.193992    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.195509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.195483    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"]
Oct 29 23:34:58.216478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.216452    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"]
Oct 29 23:34:58.285138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.285099    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.285138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.285137    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjkx\" (UniqueName: \"kubernetes.io/projected/d4d50f3d-7e8a-4a14-9864-3d9525668f82-kube-api-access-hzjkx\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.285386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.285236    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkrqq\" (UniqueName: \"kubernetes.io/projected/b33628ce-551c-4a00-bfc4-083a89ad8da1-kube-api-access-wkrqq\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.285386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.285269    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.285465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.285444    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.285499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.285475    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.285831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.285809    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.287864 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.287835    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.312498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.312473    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjkx\" (UniqueName: \"kubernetes.io/projected/d4d50f3d-7e8a-4a14-9864-3d9525668f82-kube-api-access-hzjkx\") pod \"maas-api-revoke-sub-c0bf0887-mzlf6\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.386734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.386696    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.386928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.386842    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkrqq\" (UniqueName: \"kubernetes.io/projected/b33628ce-551c-4a00-bfc4-083a89ad8da1-kube-api-access-wkrqq\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.386928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.386863    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.387458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.387439    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.389217 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.389194    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.412338 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.412271    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkrqq\" (UniqueName: \"kubernetes.io/projected/b33628ce-551c-4a00-bfc4-083a89ad8da1-kube-api-access-wkrqq\") pod \"maas-api-revoke-sub-ee104db0-d55pz\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.480538 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.480510    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:34:58.512102 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.512072    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:34:58.790277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.790245    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"]
Oct 29 23:34:58.791430 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:58.791399    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d50f3d_7e8a_4a14_9864_3d9525668f82.slice/crio-bb75df921713d981ac92e1cd19e179cdd83690757273448278cff37e70889e0a WatchSource:0}: Error finding container bb75df921713d981ac92e1cd19e179cdd83690757273448278cff37e70889e0a: Status 404 returned error can't find the container with id bb75df921713d981ac92e1cd19e179cdd83690757273448278cff37e70889e0a
Oct 29 23:34:58.832166 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:58.832138    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"]
Oct 29 23:34:58.832394 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:34:58.832369    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33628ce_551c_4a00_bfc4_083a89ad8da1.slice/crio-5206a98782a3162d052d10045dd44fb063e90881f8aa8a7b09e7b45f5e45a18e WatchSource:0}: Error finding container 5206a98782a3162d052d10045dd44fb063e90881f8aa8a7b09e7b45f5e45a18e: Status 404 returned error can't find the container with id 5206a98782a3162d052d10045dd44fb063e90881f8aa8a7b09e7b45f5e45a18e
Oct 29 23:34:59.314267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.314172    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6" event={"ID":"d4d50f3d-7e8a-4a14-9864-3d9525668f82","Type":"ContainerStarted","Data":"535f77e20901a705235d31870d308a406f7aed56b2c5611f307a99ee543f6003"}
Oct 29 23:34:59.314657 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.314271    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6" event={"ID":"d4d50f3d-7e8a-4a14-9864-3d9525668f82","Type":"ContainerStarted","Data":"bb75df921713d981ac92e1cd19e179cdd83690757273448278cff37e70889e0a"}
Oct 29 23:34:59.315922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.315891    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz" event={"ID":"b33628ce-551c-4a00-bfc4-083a89ad8da1","Type":"ContainerStarted","Data":"68e5f889bcfb6d4ca86c9d3658e86d0c1f1515bb0cda315e53826f76cdd178a2"}
Oct 29 23:34:59.316031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.315950    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz" event={"ID":"b33628ce-551c-4a00-bfc4-083a89ad8da1","Type":"ContainerStarted","Data":"5206a98782a3162d052d10045dd44fb063e90881f8aa8a7b09e7b45f5e45a18e"}
Oct 29 23:34:59.359976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.359924    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6" podStartSLOduration=1.3599096990000001 podStartE2EDuration="1.359909699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:59.359021069 +0000 UTC m=+1277.463281454" watchObservedRunningTime="2026-10-29 23:34:59.359909699 +0000 UTC m=+1277.464170078"
Oct 29 23:34:59.406402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.406350    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz" podStartSLOduration=1.406337127 podStartE2EDuration="1.406337127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:34:59.40394331 +0000 UTC m=+1277.508203696" watchObservedRunningTime="2026-10-29 23:34:59.406337127 +0000 UTC m=+1277.510597513"
Oct 29 23:34:59.768504 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.768464    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" podUID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" containerName="payload-processing" containerID="cri-o://48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11" gracePeriod=25
Oct 29 23:34:59.798425 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:34:59.798379    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" podUID="702ac16e-58f7-402f-90e7-75d68204613a" containerName="payload-pre-processing" containerID="cri-o://8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66" gracePeriod=25
Oct 29 23:35:00.026022 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:00.025931    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df1bfda-c485-4595-a210-6a5e28a80e7f" path="/var/lib/kubelet/pods/2df1bfda-c485-4595-a210-6a5e28a80e7f/volumes"
Oct 29 23:35:00.325167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:00.325071    6203 generic.go:356] "Generic (PLEG): container finished" podID="d4d50f3d-7e8a-4a14-9864-3d9525668f82" containerID="535f77e20901a705235d31870d308a406f7aed56b2c5611f307a99ee543f6003" exitCode=0
Oct 29 23:35:00.325647 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:00.325160    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6" event={"ID":"d4d50f3d-7e8a-4a14-9864-3d9525668f82","Type":"ContainerDied","Data":"535f77e20901a705235d31870d308a406f7aed56b2c5611f307a99ee543f6003"}
Oct 29 23:35:00.326752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:00.326716    6203 generic.go:356] "Generic (PLEG): container finished" podID="b33628ce-551c-4a00-bfc4-083a89ad8da1" containerID="68e5f889bcfb6d4ca86c9d3658e86d0c1f1515bb0cda315e53826f76cdd178a2" exitCode=0
Oct 29 23:35:00.326887 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:00.326756    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz" event={"ID":"b33628ce-551c-4a00-bfc4-083a89ad8da1","Type":"ContainerDied","Data":"68e5f889bcfb6d4ca86c9d3658e86d0c1f1515bb0cda315e53826f76cdd178a2"}
Oct 29 23:35:01.061784 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.061754    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:35:01.112485 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.112461    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:35:01.118085 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.118064    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:35:01.121497 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121473    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-envoy\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.121617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121575    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istiod-ca-cert\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.121678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121634    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-token\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-token\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.121678 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121666    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-podinfo\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.121799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121717    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-kube-api-access-fvrxg\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-kube-api-access-fvrxg\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.121799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121771    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-socket\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.121920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121892    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-data\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.121979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121933    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-credential-socket\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.122041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.121990    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-certs\") pod \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\" (UID: \"bc61897c-8bd9-48f7-97eb-2a4e277c72f4\") "
Oct 29 23:35:01.122101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122050    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istiod-ca-cert" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:01.122366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122287    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-credential-socket" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.122366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122305    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-data" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.122565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122376    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-socket" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.122565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122490    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-certs" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.122686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122643    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.122686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122667    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.122746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122684    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.122746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122699    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.122746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.122714    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.123664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.123647    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-envoy" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.123939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.123920    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-podinfo" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:01.124235 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.124218    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-kube-api-access-fvrxg" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "kube-api-access-fvrxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.124277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.124229    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-token" pod "bc61897c-8bd9-48f7-97eb-2a4e277c72f4" (UID: "bc61897c-8bd9-48f7-97eb-2a4e277c72f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.224102 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224063    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-envoy\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224116    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-kube-api-access-9ktpz\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-kube-api-access-9ktpz\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224140    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-podinfo\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.224302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224166    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-envoy\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.224302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224195    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-credential-socket\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.224302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224240    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istiod-ca-cert\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224361    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1138d3a-a1ba-4384-8521-920fc1b1339a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1138d3a-a1ba-4384-8521-920fc1b1339a-istiod-ca-cert\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.224583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224410    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-podinfo\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224500    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-data\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224545    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-socket\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.224788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224583    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-socket\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224654    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-credential-socket\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224695    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-token\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-token\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.224788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224754    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-credential-socket" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.224788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224775    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-data\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.225077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224809    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-token\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-token\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.225077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224845    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istiod-ca-cert" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:01.225077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224871    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-kube-api-access-mcnq9\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-kube-api-access-mcnq9\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.225077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224948    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-certs\") pod \"f1138d3a-a1ba-4384-8521-920fc1b1339a\" (UID: \"f1138d3a-a1ba-4384-8521-920fc1b1339a\") "
Oct 29 23:35:01.225077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.224981    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-certs\") pod \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\" (UID: \"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504\") "
Oct 29 23:35:01.225351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225136    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-socket" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.225351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225223    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-data" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.225351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225332    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-socket" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.225515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225405    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-credential-socket" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.225515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225428    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-certs" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.225515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225479    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-data" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.225671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225519    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1138d3a-a1ba-4384-8521-920fc1b1339a-istiod-ca-cert" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:01.225723 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225668    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-certs" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.225914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225864    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.225914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225908    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1138d3a-a1ba-4384-8521-920fc1b1339a-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225923    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225936    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225950    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225962    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225977    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.225990    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.226002    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.226016    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvrxg\" (UniqueName: \"kubernetes.io/projected/bc61897c-8bd9-48f7-97eb-2a4e277c72f4-kube-api-access-fvrxg\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.226030    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.226043    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.226056    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.226065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.226069    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.228653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.228592    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-token" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.228827 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.228753    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-envoy" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.229311 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.229274    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-podinfo" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:01.229716 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.229689    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-envoy" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:01.230073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.230016    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-kube-api-access-9ktpz" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "kube-api-access-9ktpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.230575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.230548    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-kube-api-access-mcnq9" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "kube-api-access-mcnq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.231205 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.231155    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-token" pod "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" (UID: "7d0fa53c-cb5f-4552-9d6b-fc8aebea0504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.234577 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.234553    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-podinfo" pod "f1138d3a-a1ba-4384-8521-920fc1b1339a" (UID: "f1138d3a-a1ba-4384-8521-920fc1b1339a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:01.328072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.327832    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.328072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.327866    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.328072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.327909    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.328072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.327924    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcnq9\" (UniqueName: \"kubernetes.io/projected/f1138d3a-a1ba-4384-8521-920fc1b1339a-kube-api-access-mcnq9\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.328072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.327937    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.328072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.327953    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ktpz\" (UniqueName: \"kubernetes.io/projected/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504-kube-api-access-9ktpz\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.328072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.327974    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.329153 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.328148    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1138d3a-a1ba-4384-8521-920fc1b1339a-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.336632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.336604    6203 generic.go:356] "Generic (PLEG): container finished" podID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerID="6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135" exitCode=0
Oct 29 23:35:01.336772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.336658    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"
Oct 29 23:35:01.336772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.336683    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" event={"ID":"bc61897c-8bd9-48f7-97eb-2a4e277c72f4","Type":"ContainerDied","Data":"6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135"}
Oct 29 23:35:01.336772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.336753    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr" event={"ID":"bc61897c-8bd9-48f7-97eb-2a4e277c72f4","Type":"ContainerDied","Data":"73979831a9fc748226275b7cf0ec162a3e0565bcff3d6dc435a0dd6fec6844a3"}
Oct 29 23:35:01.336946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.336787    6203 scope.go:122] "RemoveContainer" containerID="6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135"
Oct 29 23:35:01.338173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.338147    6203 generic.go:356] "Generic (PLEG): container finished" podID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerID="cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463" exitCode=0
Oct 29 23:35:01.338280 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.338189    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"
Oct 29 23:35:01.338280 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.338247    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" event={"ID":"f1138d3a-a1ba-4384-8521-920fc1b1339a","Type":"ContainerDied","Data":"cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463"}
Oct 29 23:35:01.338420 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.338299    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6" event={"ID":"f1138d3a-a1ba-4384-8521-920fc1b1339a","Type":"ContainerDied","Data":"54f0b37aa4282d41e61787f9ac15a2bec568cda828976b5ef584377a54b343ee"}
Oct 29 23:35:01.340146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.340110    6203 generic.go:356] "Generic (PLEG): container finished" podID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerID="5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7" exitCode=0
Oct 29 23:35:01.340146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.340146    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"
Oct 29 23:35:01.340323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.340233    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" event={"ID":"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504","Type":"ContainerDied","Data":"5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7"}
Oct 29 23:35:01.340323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.340275    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh" event={"ID":"7d0fa53c-cb5f-4552-9d6b-fc8aebea0504","Type":"ContainerDied","Data":"3b764a98cfa76261424147afe9152269ecc7c3c0f415fc56969e81cdc0bd4fe7"}
Oct 29 23:35:01.354988 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.354957    6203 scope.go:122] "RemoveContainer" containerID="6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135"
Oct 29 23:35:01.355307 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:01.355285    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135\": container with ID starting with 6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135 not found: ID does not exist" containerID="6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135"
Oct 29 23:35:01.355402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.355314    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135"} err="failed to get container status \"6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135\": rpc error: code = NotFound desc = could not find container \"6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135\": container with ID starting with 6301cf2ccc0dee1fd3a6063d7e008c419b27a1c620dd9643aff4a5bb9da21135 not found: ID does not exist"
Oct 29 23:35:01.355402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.355332    6203 scope.go:122] "RemoveContainer" containerID="cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463"
Oct 29 23:35:01.369491 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.369463    6203 scope.go:122] "RemoveContainer" containerID="cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463"
Oct 29 23:35:01.369827 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:01.369799    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463\": container with ID starting with cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463 not found: ID does not exist" containerID="cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463"
Oct 29 23:35:01.369892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.369842    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463"} err="failed to get container status \"cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463\": rpc error: code = NotFound desc = could not find container \"cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463\": container with ID starting with cb0ca1e1e128b4ebd565785d3b417b0aee166f97268f8777d3555137040e1463 not found: ID does not exist"
Oct 29 23:35:01.369892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.369870    6203 scope.go:122] "RemoveContainer" containerID="5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7"
Oct 29 23:35:01.382984 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.382955    6203 scope.go:122] "RemoveContainer" containerID="5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7"
Oct 29 23:35:01.383279 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:01.383256    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7\": container with ID starting with 5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7 not found: ID does not exist" containerID="5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7"
Oct 29 23:35:01.383367 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.383290    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7"} err="failed to get container status \"5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7\": rpc error: code = NotFound desc = could not find container \"5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7\": container with ID starting with 5e3f2fc0353ea78b76fa00d95835805b92e8caa844ee6f80ade6bbf2fe1b2bd7 not found: ID does not exist"
Oct 29 23:35:01.398706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.398665    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"]
Oct 29 23:35:01.409742 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.408305    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-6fxzh"]
Oct 29 23:35:01.446748 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.446714    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"]
Oct 29 23:35:01.450620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.450590    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-7rftr"]
Oct 29 23:35:01.487430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.487399    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:35:01.493971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.493940    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"]
Oct 29 23:35:01.495968 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.495938    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2a53c9-openshift-default-6fdbfb78c7-tb2n6"]
Oct 29 23:35:01.519662 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.519640    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:35:01.531114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.531077    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-xks-ca\") pod \"b33628ce-551c-4a00-bfc4-083a89ad8da1\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") "
Oct 29 23:35:01.531250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.531204    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33628ce-551c-4a00-bfc4-083a89ad8da1-kube-api-access-wkrqq\" (UniqueName: \"kubernetes.io/projected/b33628ce-551c-4a00-bfc4-083a89ad8da1-kube-api-access-wkrqq\") pod \"b33628ce-551c-4a00-bfc4-083a89ad8da1\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") "
Oct 29 23:35:01.531495 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.531472    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-openshift-ca\") pod \"b33628ce-551c-4a00-bfc4-083a89ad8da1\" (UID: \"b33628ce-551c-4a00-bfc4-083a89ad8da1\") "
Oct 29 23:35:01.531924 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.531889    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-openshift-ca" pod "b33628ce-551c-4a00-bfc4-083a89ad8da1" (UID: "b33628ce-551c-4a00-bfc4-083a89ad8da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:01.533190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.533174    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-xks-ca" pod "b33628ce-551c-4a00-bfc4-083a89ad8da1" (UID: "b33628ce-551c-4a00-bfc4-083a89ad8da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:01.533335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.533313    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33628ce-551c-4a00-bfc4-083a89ad8da1-kube-api-access-wkrqq" pod "b33628ce-551c-4a00-bfc4-083a89ad8da1" (UID: "b33628ce-551c-4a00-bfc4-083a89ad8da1"). InnerVolumeSpecName "kube-api-access-wkrqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.633031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.632946    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-xks-ca\") pod \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") "
Oct 29 23:35:01.633183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.633057    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-openshift-ca\") pod \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") "
Oct 29 23:35:01.633265 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.633224    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d50f3d-7e8a-4a14-9864-3d9525668f82-kube-api-access-hzjkx\" (UniqueName: \"kubernetes.io/projected/d4d50f3d-7e8a-4a14-9864-3d9525668f82-kube-api-access-hzjkx\") pod \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\" (UID: \"d4d50f3d-7e8a-4a14-9864-3d9525668f82\") "
Oct 29 23:35:01.633416 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.633386    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-openshift-ca" pod "d4d50f3d-7e8a-4a14-9864-3d9525668f82" (UID: "d4d50f3d-7e8a-4a14-9864-3d9525668f82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:01.633718 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.633699    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.633800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.633724    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.633800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.633738    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b33628ce-551c-4a00-bfc4-083a89ad8da1-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.633800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.633752    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkrqq\" (UniqueName: \"kubernetes.io/projected/b33628ce-551c-4a00-bfc4-083a89ad8da1-kube-api-access-wkrqq\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.635124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.635102    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-xks-ca" pod "d4d50f3d-7e8a-4a14-9864-3d9525668f82" (UID: "d4d50f3d-7e8a-4a14-9864-3d9525668f82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:01.635199 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.635184    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d50f3d-7e8a-4a14-9864-3d9525668f82-kube-api-access-hzjkx" pod "d4d50f3d-7e8a-4a14-9864-3d9525668f82" (UID: "d4d50f3d-7e8a-4a14-9864-3d9525668f82"). InnerVolumeSpecName "kube-api-access-hzjkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:01.735162 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.735122    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjkx\" (UniqueName: \"kubernetes.io/projected/d4d50f3d-7e8a-4a14-9864-3d9525668f82-kube-api-access-hzjkx\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:01.735162 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:01.735155    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d50f3d-7e8a-4a14-9864-3d9525668f82-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:02.019237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.019202    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" path="/var/lib/kubelet/pods/7d0fa53c-cb5f-4552-9d6b-fc8aebea0504/volumes"
Oct 29 23:35:02.020066 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.020032    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" path="/var/lib/kubelet/pods/bc61897c-8bd9-48f7-97eb-2a4e277c72f4/volumes"
Oct 29 23:35:02.020483 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.020469    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" path="/var/lib/kubelet/pods/f1138d3a-a1ba-4384-8521-920fc1b1339a/volumes"
Oct 29 23:35:02.351308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.351213    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6" event={"ID":"d4d50f3d-7e8a-4a14-9864-3d9525668f82","Type":"ContainerDied","Data":"bb75df921713d981ac92e1cd19e179cdd83690757273448278cff37e70889e0a"}
Oct 29 23:35:02.351308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.351269    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb75df921713d981ac92e1cd19e179cdd83690757273448278cff37e70889e0a"
Oct 29 23:35:02.351308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.351222    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"
Oct 29 23:35:02.352983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.352962    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"
Oct 29 23:35:02.353130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.353006    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz" event={"ID":"b33628ce-551c-4a00-bfc4-083a89ad8da1","Type":"ContainerDied","Data":"5206a98782a3162d052d10045dd44fb063e90881f8aa8a7b09e7b45f5e45a18e"}
Oct 29 23:35:02.353130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:02.353046    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5206a98782a3162d052d10045dd44fb063e90881f8aa8a7b09e7b45f5e45a18e"
Oct 29 23:35:03.541722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:03.541680    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"]
Oct 29 23:35:04.067438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.067401    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"]
Oct 29 23:35:04.067644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.067450    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" containerID="cri-o://5632ad55083ef383e02aa9cce91176927f41d511516d6991adc5687f23d29619" gracePeriod=45
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068068    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b33628ce-551c-4a00-bfc4-083a89ad8da1" containerName="revoke-keys"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068083    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33628ce-551c-4a00-bfc4-083a89ad8da1" containerName="revoke-keys"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068099    6203 cpu_manager.go:414] "containerMap: removing container" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068103    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068121    6203 cpu_manager.go:414] "containerMap: removing container" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068128    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068142    6203 cpu_manager.go:414] "containerMap: removing container" podUID="d4d50f3d-7e8a-4a14-9864-3d9525668f82" containerName="revoke-keys"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068146    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d50f3d-7e8a-4a14-9864-3d9525668f82" containerName="revoke-keys"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068172    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy"
Oct 29 23:35:04.068196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068177    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy"
Oct 29 23:35:04.068602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068265    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc61897c-8bd9-48f7-97eb-2a4e277c72f4" containerName="istio-proxy"
Oct 29 23:35:04.068602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068280    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d50f3d-7e8a-4a14-9864-3d9525668f82" containerName="revoke-keys"
Oct 29 23:35:04.068602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068291    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0fa53c-cb5f-4552-9d6b-fc8aebea0504" containerName="istio-proxy"
Oct 29 23:35:04.068602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068299    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33628ce-551c-4a00-bfc4-083a89ad8da1" containerName="revoke-keys"
Oct 29 23:35:04.068602 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.068305    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1138d3a-a1ba-4384-8521-920fc1b1339a" containerName="istio-proxy"
Oct 29 23:35:04.073327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.073282    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.085643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.085575    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main" containerID="cri-o://5b8d6766fbc885fde5bbd3b9457b3e8b713601a3dda001f8718f232e3d5530be" gracePeriod=45
Oct 29 23:35:04.091702 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.091679    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:35:04.092706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.092683    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"]
Oct 29 23:35:04.093071 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.093046    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:35:04.093185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.093162    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:35:04.099163 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.099132    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"]
Oct 29 23:35:04.099267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.099249    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.117078 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.117047    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"]
Oct 29 23:35:04.181590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.181552    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.182006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.181983    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.182481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.182178    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.182481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.182335    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6f5m\" (UniqueName: \"kubernetes.io/projected/9805adb0-7e4c-4667-95f8-05754ba79910-kube-api-access-c6f5m\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.182481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.182392    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7xn\" (UniqueName: \"kubernetes.io/projected/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-kube-api-access-7g7xn\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.185505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.185475    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.191724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.191663    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.290272    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.290418    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.290449    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.290476    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.290561    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6f5m\" (UniqueName: \"kubernetes.io/projected/9805adb0-7e4c-4667-95f8-05754ba79910-kube-api-access-c6f5m\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.290617    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7xn\" (UniqueName: \"kubernetes.io/projected/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-kube-api-access-7g7xn\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.291968    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.292510    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": dial tcp 10.134.0.101:8000: connect: connection refused"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.292587    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.299916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.295485    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.304410 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.304168    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.322029 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.321928    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7xn\" (UniqueName: \"kubernetes.io/projected/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-kube-api-access-7g7xn\") pod \"maas-api-revoke-sub-efd0cd8a-6ddwd\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.324498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.324451    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6f5m\" (UniqueName: \"kubernetes.io/projected/9805adb0-7e4c-4667-95f8-05754ba79910-kube-api-access-c6f5m\") pod \"maas-api-revoke-sub-04fde25c-cd8dd\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.375077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.375042    6203 generic.go:356] "Generic (PLEG): container finished" podID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerID="5632ad55083ef383e02aa9cce91176927f41d511516d6991adc5687f23d29619" exitCode=0
Oct 29 23:35:04.375232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.375124    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" event={"ID":"087a38a6-8300-4b22-8966-e3d0e18fb2fb","Type":"ContainerDied","Data":"5632ad55083ef383e02aa9cce91176927f41d511516d6991adc5687f23d29619"}
Oct 29 23:35:04.377147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.377117    6203 generic.go:356] "Generic (PLEG): container finished" podID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerID="5b8d6766fbc885fde5bbd3b9457b3e8b713601a3dda001f8718f232e3d5530be" exitCode=0
Oct 29 23:35:04.377273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.377194    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" event={"ID":"a5fcc36f-6010-4bea-874c-dcc395e52ff1","Type":"ContainerDied","Data":"5b8d6766fbc885fde5bbd3b9457b3e8b713601a3dda001f8718f232e3d5530be"}
Oct 29 23:35:04.382825 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.382795    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:35:04.411676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.411651    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:35:04.492488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492447    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tls-certs\") pod \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") "
Oct 29 23:35:04.492665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492536    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087a38a6-8300-4b22-8966-e3d0e18fb2fb-kube-api-access-kcn9x\" (UniqueName: \"kubernetes.io/projected/087a38a6-8300-4b22-8966-e3d0e18fb2fb-kube-api-access-kcn9x\") pod \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") "
Oct 29 23:35:04.492665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492598    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-home\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-home\") pod \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") "
Oct 29 23:35:04.492665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492646    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tmp-dir\") pod \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") "
Oct 29 23:35:04.492846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492721    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-dshm\") pod \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") "
Oct 29 23:35:04.492846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492767    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tls-certs\") pod \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") "
Oct 29 23:35:04.492846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492800    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-dshm\") pod \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") "
Oct 29 23:35:04.493054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492846    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tmp-dir\") pod \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") "
Oct 29 23:35:04.493054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492923    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-home\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-home\") pod \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") "
Oct 29 23:35:04.493054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.492977    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-model-cache\") pod \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") "
Oct 29 23:35:04.493054 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.493021    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-model-cache\") pod \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\" (UID: \"087a38a6-8300-4b22-8966-e3d0e18fb2fb\") "
Oct 29 23:35:04.494055 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.493720    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fcc36f-6010-4bea-874c-dcc395e52ff1-kube-api-access-cqjn5\" (UniqueName: \"kubernetes.io/projected/a5fcc36f-6010-4bea-874c-dcc395e52ff1-kube-api-access-cqjn5\") pod \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\" (UID: \"a5fcc36f-6010-4bea-874c-dcc395e52ff1\") "
Oct 29 23:35:04.496316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.496285    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-home" pod "087a38a6-8300-4b22-8966-e3d0e18fb2fb" (UID: "087a38a6-8300-4b22-8966-e3d0e18fb2fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.496435 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.496405    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-model-cache" pod "a5fcc36f-6010-4bea-874c-dcc395e52ff1" (UID: "a5fcc36f-6010-4bea-874c-dcc395e52ff1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.496965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.496595    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tmp-dir" pod "a5fcc36f-6010-4bea-874c-dcc395e52ff1" (UID: "a5fcc36f-6010-4bea-874c-dcc395e52ff1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.496965 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.496860    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-model-cache" pod "087a38a6-8300-4b22-8966-e3d0e18fb2fb" (UID: "087a38a6-8300-4b22-8966-e3d0e18fb2fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.498344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.498312    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tmp-dir" pod "087a38a6-8300-4b22-8966-e3d0e18fb2fb" (UID: "087a38a6-8300-4b22-8966-e3d0e18fb2fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.503903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.500733    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-home" pod "a5fcc36f-6010-4bea-874c-dcc395e52ff1" (UID: "a5fcc36f-6010-4bea-874c-dcc395e52ff1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.505207 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.505180    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tls-certs" pod "a5fcc36f-6010-4bea-874c-dcc395e52ff1" (UID: "a5fcc36f-6010-4bea-874c-dcc395e52ff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:04.505372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.505353    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tls-certs" pod "087a38a6-8300-4b22-8966-e3d0e18fb2fb" (UID: "087a38a6-8300-4b22-8966-e3d0e18fb2fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:04.506697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.506672    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-dshm" pod "a5fcc36f-6010-4bea-874c-dcc395e52ff1" (UID: "a5fcc36f-6010-4bea-874c-dcc395e52ff1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.507123 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.507083    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fcc36f-6010-4bea-874c-dcc395e52ff1-kube-api-access-cqjn5" pod "a5fcc36f-6010-4bea-874c-dcc395e52ff1" (UID: "a5fcc36f-6010-4bea-874c-dcc395e52ff1"). InnerVolumeSpecName "kube-api-access-cqjn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:04.507252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.507228    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-dshm" pod "087a38a6-8300-4b22-8966-e3d0e18fb2fb" (UID: "087a38a6-8300-4b22-8966-e3d0e18fb2fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:04.507417 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.507394    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087a38a6-8300-4b22-8966-e3d0e18fb2fb-kube-api-access-kcn9x" pod "087a38a6-8300-4b22-8966-e3d0e18fb2fb" (UID: "087a38a6-8300-4b22-8966-e3d0e18fb2fb"). InnerVolumeSpecName "kube-api-access-kcn9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:04.516676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.516648    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:04.526349 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.526291    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596249    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596282    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596297    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596310    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fcc36f-6010-4bea-874c-dcc395e52ff1-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596323    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596336    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqjn5\" (UniqueName: \"kubernetes.io/projected/a5fcc36f-6010-4bea-874c-dcc395e52ff1-kube-api-access-cqjn5\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596349    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fcc36f-6010-4bea-874c-dcc395e52ff1-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596366    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcn9x\" (UniqueName: \"kubernetes.io/projected/087a38a6-8300-4b22-8966-e3d0e18fb2fb-kube-api-access-kcn9x\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596379    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596391    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596403    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/087a38a6-8300-4b22-8966-e3d0e18fb2fb-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.596437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.596415    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/087a38a6-8300-4b22-8966-e3d0e18fb2fb-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:04.836041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.836014    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"]
Oct 29 23:35:04.836377 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:35:04.836353    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4f0854_95f0_4f11_9419_b5cefd13a9c0.slice/crio-5c4f74cb40c5f4716cd7b1ba5ffd86e4253103138562e5a9f65f144477e7c198 WatchSource:0}: Error finding container 5c4f74cb40c5f4716cd7b1ba5ffd86e4253103138562e5a9f65f144477e7c198: Status 404 returned error can't find the container with id 5c4f74cb40c5f4716cd7b1ba5ffd86e4253103138562e5a9f65f144477e7c198
Oct 29 23:35:04.863067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:04.863041    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"]
Oct 29 23:35:04.867263 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:35:04.867237    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9805adb0_7e4c_4667_95f8_05754ba79910.slice/crio-b44c9178c0918f3c239346a18486e98d8099678c5c81590a2b98ae88793dc988 WatchSource:0}: Error finding container b44c9178c0918f3c239346a18486e98d8099678c5c81590a2b98ae88793dc988: Status 404 returned error can't find the container with id b44c9178c0918f3c239346a18486e98d8099678c5c81590a2b98ae88793dc988
Oct 29 23:35:05.396062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.395541    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs" event={"ID":"087a38a6-8300-4b22-8966-e3d0e18fb2fb","Type":"ContainerDied","Data":"6478849e347699aa3077ab630cd26eb4af737a801e10517f5a65286e65a7e1dd"}
Oct 29 23:35:05.396062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.395750    6203 scope.go:122] "RemoveContainer" containerID="5632ad55083ef383e02aa9cce91176927f41d511516d6991adc5687f23d29619"
Oct 29 23:35:05.396062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.396001    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"
Oct 29 23:35:05.408153 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.408128    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"
Oct 29 23:35:05.408345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.408258    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv" event={"ID":"a5fcc36f-6010-4bea-874c-dcc395e52ff1","Type":"ContainerDied","Data":"b4ce7553d21b3cff0848c7308ab8bd3021b3a99af17fde4039fe61a606012ca8"}
Oct 29 23:35:05.411244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.411213    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd" event={"ID":"9805adb0-7e4c-4667-95f8-05754ba79910","Type":"ContainerStarted","Data":"1bd670ba5a328e88abf23bd5cc1050deb9ff7390b70860815638c344edf42bc7"}
Oct 29 23:35:05.411369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.411300    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd" event={"ID":"9805adb0-7e4c-4667-95f8-05754ba79910","Type":"ContainerStarted","Data":"b44c9178c0918f3c239346a18486e98d8099678c5c81590a2b98ae88793dc988"}
Oct 29 23:35:05.413836 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.413807    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd" event={"ID":"4b4f0854-95f0-4f11-9419-b5cefd13a9c0","Type":"ContainerStarted","Data":"11876eaf8c49f065a9b27cf87a7ff4d798e86f1b23d461ce66d0f15efeeb73df"}
Oct 29 23:35:05.414176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.414156    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd" event={"ID":"4b4f0854-95f0-4f11-9419-b5cefd13a9c0","Type":"ContainerStarted","Data":"5c4f74cb40c5f4716cd7b1ba5ffd86e4253103138562e5a9f65f144477e7c198"}
Oct 29 23:35:05.423245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.423221    6203 scope.go:122] "RemoveContainer" containerID="5b8d6766fbc885fde5bbd3b9457b3e8b713601a3dda001f8718f232e3d5530be"
Oct 29 23:35:05.490018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.489961    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd" podStartSLOduration=1.489941945 podStartE2EDuration="1.489941945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:05.471941585 +0000 UTC m=+1283.576201971" watchObservedRunningTime="2026-10-29 23:35:05.489941945 +0000 UTC m=+1283.594202332"
Oct 29 23:35:05.542423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.542353    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd" podStartSLOduration=1.542329036 podStartE2EDuration="1.542329036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:05.535934138 +0000 UTC m=+1283.640194525" watchObservedRunningTime="2026-10-29 23:35:05.542329036 +0000 UTC m=+1283.646589427"
Oct 29 23:35:05.585012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.584968    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"]
Oct 29 23:35:05.591309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.591274    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/distinct-w2-ae1364-kserve-bf5fcb855-zcxvs"]
Oct 29 23:35:05.625603 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.625562    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"]
Oct 29 23:35:05.633670 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:05.633637    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ae1364/simulator-w2-ae1364-kserve-dbd84cf97-6bxcv"]
Oct 29 23:35:06.020893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:06.020844    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" path="/var/lib/kubelet/pods/087a38a6-8300-4b22-8966-e3d0e18fb2fb/volumes"
Oct 29 23:35:06.021372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:06.021353    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" path="/var/lib/kubelet/pods/a5fcc36f-6010-4bea-874c-dcc395e52ff1/volumes"
Oct 29 23:35:06.422723 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:06.422682    6203 generic.go:356] "Generic (PLEG): container finished" podID="9805adb0-7e4c-4667-95f8-05754ba79910" containerID="1bd670ba5a328e88abf23bd5cc1050deb9ff7390b70860815638c344edf42bc7" exitCode=0
Oct 29 23:35:06.422892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:06.422769    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd" event={"ID":"9805adb0-7e4c-4667-95f8-05754ba79910","Type":"ContainerDied","Data":"1bd670ba5a328e88abf23bd5cc1050deb9ff7390b70860815638c344edf42bc7"}
Oct 29 23:35:06.424402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:06.424379    6203 generic.go:356] "Generic (PLEG): container finished" podID="4b4f0854-95f0-4f11-9419-b5cefd13a9c0" containerID="11876eaf8c49f065a9b27cf87a7ff4d798e86f1b23d461ce66d0f15efeeb73df" exitCode=0
Oct 29 23:35:06.424511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:06.424465    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd" event={"ID":"4b4f0854-95f0-4f11-9419-b5cefd13a9c0","Type":"ContainerDied","Data":"11876eaf8c49f065a9b27cf87a7ff4d798e86f1b23d461ce66d0f15efeeb73df"}
Oct 29 23:35:07.602145 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.601758    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:07.626388 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.626363    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:07.666147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.666112    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-openshift-ca\") pod \"9805adb0-7e4c-4667-95f8-05754ba79910\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") "
Oct 29 23:35:07.666318 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.666179    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9805adb0-7e4c-4667-95f8-05754ba79910-kube-api-access-c6f5m\" (UniqueName: \"kubernetes.io/projected/9805adb0-7e4c-4667-95f8-05754ba79910-kube-api-access-c6f5m\") pod \"9805adb0-7e4c-4667-95f8-05754ba79910\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") "
Oct 29 23:35:07.666318 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.666283    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-xks-ca\") pod \"9805adb0-7e4c-4667-95f8-05754ba79910\" (UID: \"9805adb0-7e4c-4667-95f8-05754ba79910\") "
Oct 29 23:35:07.666590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.666563    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-openshift-ca" pod "9805adb0-7e4c-4667-95f8-05754ba79910" (UID: "9805adb0-7e4c-4667-95f8-05754ba79910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:07.666943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.666919    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:07.668467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.668434    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-xks-ca" pod "9805adb0-7e4c-4667-95f8-05754ba79910" (UID: "9805adb0-7e4c-4667-95f8-05754ba79910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:07.668585 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.668565    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9805adb0-7e4c-4667-95f8-05754ba79910-kube-api-access-c6f5m" pod "9805adb0-7e4c-4667-95f8-05754ba79910" (UID: "9805adb0-7e4c-4667-95f8-05754ba79910"). InnerVolumeSpecName "kube-api-access-c6f5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:07.768299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.768220    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-openshift-ca\") pod \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") "
Oct 29 23:35:07.768299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.768272    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-kube-api-access-7g7xn\" (UniqueName: \"kubernetes.io/projected/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-kube-api-access-7g7xn\") pod \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") "
Oct 29 23:35:07.768499 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.768311    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-xks-ca\") pod \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\" (UID: \"4b4f0854-95f0-4f11-9419-b5cefd13a9c0\") "
Oct 29 23:35:07.768666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.768643    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-openshift-ca" pod "4b4f0854-95f0-4f11-9419-b5cefd13a9c0" (UID: "4b4f0854-95f0-4f11-9419-b5cefd13a9c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:07.768793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.768780    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:07.768831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.768796    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6f5m\" (UniqueName: \"kubernetes.io/projected/9805adb0-7e4c-4667-95f8-05754ba79910-kube-api-access-c6f5m\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:07.768831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.768808    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9805adb0-7e4c-4667-95f8-05754ba79910-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:07.770399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.770374    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-xks-ca" pod "4b4f0854-95f0-4f11-9419-b5cefd13a9c0" (UID: "4b4f0854-95f0-4f11-9419-b5cefd13a9c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:07.770464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.770402    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-kube-api-access-7g7xn" pod "4b4f0854-95f0-4f11-9419-b5cefd13a9c0" (UID: "4b4f0854-95f0-4f11-9419-b5cefd13a9c0"). InnerVolumeSpecName "kube-api-access-7g7xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:07.870296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.870258    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:07.870296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:07.870290    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g7xn\" (UniqueName: \"kubernetes.io/projected/4b4f0854-95f0-4f11-9419-b5cefd13a9c0-kube-api-access-7g7xn\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:08.422374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.422342    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" podUID="702ac16e-58f7-402f-90e7-75d68204613a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:08.422517 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.422342    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" podUID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:08.443333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.443307    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"
Oct 29 23:35:08.443505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.443328    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd" event={"ID":"9805adb0-7e4c-4667-95f8-05754ba79910","Type":"ContainerDied","Data":"b44c9178c0918f3c239346a18486e98d8099678c5c81590a2b98ae88793dc988"}
Oct 29 23:35:08.443505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.443383    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b44c9178c0918f3c239346a18486e98d8099678c5c81590a2b98ae88793dc988"
Oct 29 23:35:08.444774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.444754    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"
Oct 29 23:35:08.444774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.444766    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd" event={"ID":"4b4f0854-95f0-4f11-9419-b5cefd13a9c0","Type":"ContainerDied","Data":"5c4f74cb40c5f4716cd7b1ba5ffd86e4253103138562e5a9f65f144477e7c198"}
Oct 29 23:35:08.444937 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.444817    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4f74cb40c5f4716cd7b1ba5ffd86e4253103138562e5a9f65f144477e7c198"
Oct 29 23:35:08.542467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:08.542404    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" podUID="71ebd8bf-7fff-47e9-b706-6baf7f325180" containerName="payload-processing" containerID="cri-o://e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3" gracePeriod=25
Oct 29 23:35:09.411596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:09.411562    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"]
Oct 29 23:35:09.426481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:09.426443    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" podUID="71ebd8bf-7fff-47e9-b706-6baf7f325180" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:09.426481 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:09.426463    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ae1364-6r2sh"]
Oct 29 23:35:09.994904 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:09.994076    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:35:10.023410 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:10.023371    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a854c1eb-67ea-4b7c-9dd2-9d02595b53f6" path="/var/lib/kubelet/pods/a854c1eb-67ea-4b7c-9dd2-9d02595b53f6/volumes"
Oct 29 23:35:11.390000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.389966    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"]
Oct 29 23:35:11.390488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390475    6203 cpu_manager.go:414] "containerMap: removing container" podUID="9805adb0-7e4c-4667-95f8-05754ba79910" containerName="revoke-keys"
Oct 29 23:35:11.390537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390489    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9805adb0-7e4c-4667-95f8-05754ba79910" containerName="revoke-keys"
Oct 29 23:35:11.390537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390523    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main"
Oct 29 23:35:11.390537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390528    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main"
Oct 29 23:35:11.390629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390549    6203 cpu_manager.go:414] "containerMap: removing container" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main"
Oct 29 23:35:11.390629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390563    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main"
Oct 29 23:35:11.390629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390574    6203 cpu_manager.go:414] "containerMap: removing container" podUID="4b4f0854-95f0-4f11-9419-b5cefd13a9c0" containerName="revoke-keys"
Oct 29 23:35:11.390629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390581    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4f0854-95f0-4f11-9419-b5cefd13a9c0" containerName="revoke-keys"
Oct 29 23:35:11.390752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390655    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fcc36f-6010-4bea-874c-dcc395e52ff1" containerName="main"
Oct 29 23:35:11.390752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390667    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="087a38a6-8300-4b22-8966-e3d0e18fb2fb" containerName="main"
Oct 29 23:35:11.390752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390674    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4f0854-95f0-4f11-9419-b5cefd13a9c0" containerName="revoke-keys"
Oct 29 23:35:11.390752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.390683    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="9805adb0-7e4c-4667-95f8-05754ba79910" containerName="revoke-keys"
Oct 29 23:35:11.395341 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.395321    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.416233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.416204    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"]
Oct 29 23:35:11.424355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.424324    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.424524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.424475    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmg7\" (UniqueName: \"kubernetes.io/projected/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-kube-api-access-6hmg7\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.424594 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.424522    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.426794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.426766    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:35:11.426794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.426766    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:35:11.427064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.426862    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:35:11.428038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.428017    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:35:11.526216 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.526184    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.526395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.526275    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmg7\" (UniqueName: \"kubernetes.io/projected/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-kube-api-access-6hmg7\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.526395 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.526311    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.527017 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.526992    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.528521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.528503    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.554065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.554035    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmg7\" (UniqueName: \"kubernetes.io/projected/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-kube-api-access-6hmg7\") pod \"maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:11.708632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:11.708597    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:12.013227 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:35:12.013194    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50efe9a_cfd5_490d_aed9_1fe2d4cf362f.slice/crio-09a81130fe72a6aecf473a4485f2e66d9fff82b1b1c2de3fd9c1d050424bfd57 WatchSource:0}: Error finding container 09a81130fe72a6aecf473a4485f2e66d9fff82b1b1c2de3fd9c1d050424bfd57: Status 404 returned error can't find the container with id 09a81130fe72a6aecf473a4485f2e66d9fff82b1b1c2de3fd9c1d050424bfd57
Oct 29 23:35:12.022853 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:12.022820    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"]
Oct 29 23:35:12.510853 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:12.510814    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv" event={"ID":"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f","Type":"ContainerStarted","Data":"6c7cd96b65dfa76ac9153147f9448e03c3599a57d2a24b1d0cbff82ca61aafd9"}
Oct 29 23:35:12.511257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:12.510886    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv" event={"ID":"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f","Type":"ContainerStarted","Data":"09a81130fe72a6aecf473a4485f2e66d9fff82b1b1c2de3fd9c1d050424bfd57"}
Oct 29 23:35:12.564555 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:12.564482    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv" podStartSLOduration=1.564464245 podStartE2EDuration="1.564464245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:35:12.562985647 +0000 UTC m=+1290.667246033" watchObservedRunningTime="2026-10-29 23:35:12.564464245 +0000 UTC m=+1290.668724632"
Oct 29 23:35:12.836805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:12.836752    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" containerID="cri-o://f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3" gracePeriod=45
Oct 29 23:35:12.847327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:12.847290    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main" containerID="cri-o://568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc" gracePeriod=45
Oct 29 23:35:12.881363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:12.881315    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": dial tcp 10.134.0.107:8000: connect: connection refused"
Oct 29 23:35:13.125169 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.125143    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:35:13.138316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.138288    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:35:13.251272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.251239    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-dshm\") pod \"a93027eb-7194-4f70-b09c-d7a767d1715a\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") "
Oct 29 23:35:13.251903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.251865    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tls-certs\") pod \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") "
Oct 29 23:35:13.252113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252095    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tmp-dir\") pod \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") "
Oct 29 23:35:13.252239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252222    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-model-cache\") pod \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") "
Oct 29 23:35:13.252427 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252412    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d706de1-efaf-4c90-b710-eab26d3ed7f8-kube-api-access-bcrxz\" (UniqueName: \"kubernetes.io/projected/6d706de1-efaf-4c90-b710-eab26d3ed7f8-kube-api-access-bcrxz\") pod \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252530    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-model-cache" pod "6d706de1-efaf-4c90-b710-eab26d3ed7f8" (UID: "6d706de1-efaf-4c90-b710-eab26d3ed7f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252557    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93027eb-7194-4f70-b09c-d7a767d1715a-kube-api-access-vpll5\" (UniqueName: \"kubernetes.io/projected/a93027eb-7194-4f70-b09c-d7a767d1715a-kube-api-access-vpll5\") pod \"a93027eb-7194-4f70-b09c-d7a767d1715a\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252637    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-home\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-home\") pod \"a93027eb-7194-4f70-b09c-d7a767d1715a\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252674    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93027eb-7194-4f70-b09c-d7a767d1715a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a93027eb-7194-4f70-b09c-d7a767d1715a-tls-certs\") pod \"a93027eb-7194-4f70-b09c-d7a767d1715a\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252834    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-model-cache\") pod \"a93027eb-7194-4f70-b09c-d7a767d1715a\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252926    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-home\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-home\") pod \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.252973    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-tmp-dir\") pod \"a93027eb-7194-4f70-b09c-d7a767d1715a\" (UID: \"a93027eb-7194-4f70-b09c-d7a767d1715a\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.253020    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-dshm\") pod \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\" (UID: \"6d706de1-efaf-4c90-b710-eab26d3ed7f8\") "
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.253573    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.254195    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-home" pod "a93027eb-7194-4f70-b09c-d7a767d1715a" (UID: "a93027eb-7194-4f70-b09c-d7a767d1715a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.254230    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tmp-dir" pod "6d706de1-efaf-4c90-b710-eab26d3ed7f8" (UID: "6d706de1-efaf-4c90-b710-eab26d3ed7f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.254364    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-model-cache" pod "a93027eb-7194-4f70-b09c-d7a767d1715a" (UID: "a93027eb-7194-4f70-b09c-d7a767d1715a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.254710 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.254403    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-home" pod "6d706de1-efaf-4c90-b710-eab26d3ed7f8" (UID: "6d706de1-efaf-4c90-b710-eab26d3ed7f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.255689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.255648    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-tmp-dir" pod "a93027eb-7194-4f70-b09c-d7a767d1715a" (UID: "a93027eb-7194-4f70-b09c-d7a767d1715a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.256063 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.256032    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tls-certs" pod "6d706de1-efaf-4c90-b710-eab26d3ed7f8" (UID: "6d706de1-efaf-4c90-b710-eab26d3ed7f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:13.256320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.256300    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-dshm" pod "a93027eb-7194-4f70-b09c-d7a767d1715a" (UID: "a93027eb-7194-4f70-b09c-d7a767d1715a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.256448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.256327    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93027eb-7194-4f70-b09c-d7a767d1715a-tls-certs" pod "a93027eb-7194-4f70-b09c-d7a767d1715a" (UID: "a93027eb-7194-4f70-b09c-d7a767d1715a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:13.256556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.256475    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d706de1-efaf-4c90-b710-eab26d3ed7f8-kube-api-access-bcrxz" pod "6d706de1-efaf-4c90-b710-eab26d3ed7f8" (UID: "6d706de1-efaf-4c90-b710-eab26d3ed7f8"). InnerVolumeSpecName "kube-api-access-bcrxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:13.257545 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.257517    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-dshm" pod "6d706de1-efaf-4c90-b710-eab26d3ed7f8" (UID: "6d706de1-efaf-4c90-b710-eab26d3ed7f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:13.259094 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.259070    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93027eb-7194-4f70-b09c-d7a767d1715a-kube-api-access-vpll5" pod "a93027eb-7194-4f70-b09c-d7a767d1715a" (UID: "a93027eb-7194-4f70-b09c-d7a767d1715a"). InnerVolumeSpecName "kube-api-access-vpll5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354823    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354855    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354863    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354871    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354896    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354904    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354912    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d706de1-efaf-4c90-b710-eab26d3ed7f8-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354920    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrxz\" (UniqueName: \"kubernetes.io/projected/6d706de1-efaf-4c90-b710-eab26d3ed7f8-kube-api-access-bcrxz\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354929    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpll5\" (UniqueName: \"kubernetes.io/projected/a93027eb-7194-4f70-b09c-d7a767d1715a-kube-api-access-vpll5\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354937    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a93027eb-7194-4f70-b09c-d7a767d1715a-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.354942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.354945    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a93027eb-7194-4f70-b09c-d7a767d1715a-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:13.519839 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.519806    6203 generic.go:356] "Generic (PLEG): container finished" podID="b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" containerID="6c7cd96b65dfa76ac9153147f9448e03c3599a57d2a24b1d0cbff82ca61aafd9" exitCode=0
Oct 29 23:35:13.520281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.519920    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv" event={"ID":"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f","Type":"ContainerDied","Data":"6c7cd96b65dfa76ac9153147f9448e03c3599a57d2a24b1d0cbff82ca61aafd9"}
Oct 29 23:35:13.521328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.521298    6203 generic.go:356] "Generic (PLEG): container finished" podID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerID="f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3" exitCode=0
Oct 29 23:35:13.521456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.521338    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"
Oct 29 23:35:13.521456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.521387    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" event={"ID":"a93027eb-7194-4f70-b09c-d7a767d1715a","Type":"ContainerDied","Data":"f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3"}
Oct 29 23:35:13.521456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.521430    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w" event={"ID":"a93027eb-7194-4f70-b09c-d7a767d1715a","Type":"ContainerDied","Data":"f356d380fba76ee350646d4a3a052d971b05e2779c2170d8e723d622b4dc7ff8"}
Oct 29 23:35:13.521456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.521453    6203 scope.go:122] "RemoveContainer" containerID="f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3"
Oct 29 23:35:13.523074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.523051    6203 generic.go:356] "Generic (PLEG): container finished" podID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerID="568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc" exitCode=0
Oct 29 23:35:13.523202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.523102    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" event={"ID":"6d706de1-efaf-4c90-b710-eab26d3ed7f8","Type":"ContainerDied","Data":"568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc"}
Oct 29 23:35:13.523202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.523109    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"
Oct 29 23:35:13.523202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.523123    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h" event={"ID":"6d706de1-efaf-4c90-b710-eab26d3ed7f8","Type":"ContainerDied","Data":"c014f22b482d0dd826e3597d19fdf9042f6a6911500c24235398a7e1ef81318f"}
Oct 29 23:35:13.531903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.531858    6203 scope.go:122] "RemoveContainer" containerID="f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3"
Oct 29 23:35:13.532194 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:13.532174    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3\": container with ID starting with f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3 not found: ID does not exist" containerID="f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3"
Oct 29 23:35:13.532284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.532201    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3"} err="failed to get container status \"f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3\": rpc error: code = NotFound desc = could not find container \"f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3\": container with ID starting with f36a75b791602c41a3249aa788906220d37dd77f4284af82caba333bea2441e3 not found: ID does not exist"
Oct 29 23:35:13.532284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.532219    6203 scope.go:122] "RemoveContainer" containerID="568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc"
Oct 29 23:35:13.550954 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.550933    6203 scope.go:122] "RemoveContainer" containerID="568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc"
Oct 29 23:35:13.551233 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:13.551215    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc\": container with ID starting with 568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc not found: ID does not exist" containerID="568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc"
Oct 29 23:35:13.551300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.551240    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc"} err="failed to get container status \"568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc\": rpc error: code = NotFound desc = could not find container \"568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc\": container with ID starting with 568c44078e1060f79c163e539648b82b5d3cc9a6e9f59462f751b846f773decc not found: ID does not exist"
Oct 29 23:35:13.610513 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.610425    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"]
Oct 29 23:35:13.617467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.617440    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/distinct-w6-ae1784-kserve-6c9f4fb85-xr85w"]
Oct 29 23:35:13.656201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.656172    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"]
Oct 29 23:35:13.659834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:13.659813    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ae1784/premium-w6-ae1784-kserve-5584cc646-fnr8h"]
Oct 29 23:35:14.018659 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.018623    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" path="/var/lib/kubelet/pods/6d706de1-efaf-4c90-b710-eab26d3ed7f8/volumes"
Oct 29 23:35:14.019126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.019112    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" path="/var/lib/kubelet/pods/a93027eb-7194-4f70-b09c-d7a767d1715a/volumes"
Oct 29 23:35:14.673004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.672980    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:14.771025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.770990    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-xks-ca\") pod \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") "
Oct 29 23:35:14.771226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.771140    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-openshift-ca\") pod \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") "
Oct 29 23:35:14.771226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.771214    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-kube-api-access-6hmg7\" (UniqueName: \"kubernetes.io/projected/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-kube-api-access-6hmg7\") pod \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\" (UID: \"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f\") "
Oct 29 23:35:14.771561 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.771535    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-openshift-ca" pod "b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" (UID: "b50efe9a-cfd5-490d-aed9-1fe2d4cf362f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:14.771746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.771727    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:14.773279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.773251    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-xks-ca" pod "b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" (UID: "b50efe9a-cfd5-490d-aed9-1fe2d4cf362f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:14.773381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.773281    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-kube-api-access-6hmg7" pod "b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" (UID: "b50efe9a-cfd5-490d-aed9-1fe2d4cf362f"). InnerVolumeSpecName "kube-api-access-6hmg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:14.872576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.872497    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmg7\" (UniqueName: \"kubernetes.io/projected/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-kube-api-access-6hmg7\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:14.872576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:14.872523    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:15.541611 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:15.541585    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"
Oct 29 23:35:15.541782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:15.541622    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv" event={"ID":"b50efe9a-cfd5-490d-aed9-1fe2d4cf362f","Type":"ContainerDied","Data":"09a81130fe72a6aecf473a4485f2e66d9fff82b1b1c2de3fd9c1d050424bfd57"}
Oct 29 23:35:15.541782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:15.541678    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a81130fe72a6aecf473a4485f2e66d9fff82b1b1c2de3fd9c1d050424bfd57"
Oct 29 23:35:17.519800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:17.519760    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"]
Oct 29 23:35:17.520364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:17.519986    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" containerID="cri-o://4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd" gracePeriod=30
Oct 29 23:35:17.542726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:17.542668    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"]
Oct 29 23:35:17.543087 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:17.543038    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" containerID="cri-o://113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13" gracePeriod=30
Oct 29 23:35:17.547175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:17.547145    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"]
Oct 29 23:35:17.547407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:17.547371    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy" containerID="cri-o://c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26" gracePeriod=30
Oct 29 23:35:18.077324 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:18.077281    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:35:18.077533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:18.077360    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:35:18.421557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:18.421521    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" podUID="702ac16e-58f7-402f-90e7-75d68204613a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:18.421743 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:18.421522    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" podUID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:19.427074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:19.427039    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" podUID="71ebd8bf-7fff-47e9-b706-6baf7f325180" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:20.065998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.065962    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:35:20.096815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.096785    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:35:20.143465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.143431    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-plugins-config-volume\") pod \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") "
Oct 29 23:35:20.143651 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.143549    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-kube-api-access-9js22\" (UniqueName: \"kubernetes.io/projected/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-kube-api-access-9js22\") pod \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\" (UID: \"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873\") "
Oct 29 23:35:20.144141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.143897    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-plugins-config-volume" pod "a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" (UID: "a93ece91-3b4f-4cb2-a9df-1e1d4f51d873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:20.144278 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.144238    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:20.146290 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.146259    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-kube-api-access-9js22" pod "a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" (UID: "a93ece91-3b4f-4cb2-a9df-1e1d4f51d873"). InnerVolumeSpecName "kube-api-access-9js22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:20.245016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.244982    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702ac16e-58f7-402f-90e7-75d68204613a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ac16e-58f7-402f-90e7-75d68204613a-plugins-config-volume\") pod \"702ac16e-58f7-402f-90e7-75d68204613a\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") "
Oct 29 23:35:20.245228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.245062    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702ac16e-58f7-402f-90e7-75d68204613a-kube-api-access-5nprw\" (UniqueName: \"kubernetes.io/projected/702ac16e-58f7-402f-90e7-75d68204613a-kube-api-access-5nprw\") pod \"702ac16e-58f7-402f-90e7-75d68204613a\" (UID: \"702ac16e-58f7-402f-90e7-75d68204613a\") "
Oct 29 23:35:20.245458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.245430    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702ac16e-58f7-402f-90e7-75d68204613a-plugins-config-volume" pod "702ac16e-58f7-402f-90e7-75d68204613a" (UID: "702ac16e-58f7-402f-90e7-75d68204613a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:20.245614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.245596    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702ac16e-58f7-402f-90e7-75d68204613a-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:20.245671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.245621    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9js22\" (UniqueName: \"kubernetes.io/projected/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873-kube-api-access-9js22\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:20.247258 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.247233    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702ac16e-58f7-402f-90e7-75d68204613a-kube-api-access-5nprw" pod "702ac16e-58f7-402f-90e7-75d68204613a" (UID: "702ac16e-58f7-402f-90e7-75d68204613a"). InnerVolumeSpecName "kube-api-access-5nprw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:20.346785 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.346709    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nprw\" (UniqueName: \"kubernetes.io/projected/702ac16e-58f7-402f-90e7-75d68204613a-kube-api-access-5nprw\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:20.581509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.581471    6203 generic.go:356] "Generic (PLEG): container finished" podID="702ac16e-58f7-402f-90e7-75d68204613a" containerID="8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66" exitCode=0
Oct 29 23:35:20.581961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.581526    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"
Oct 29 23:35:20.581961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.581561    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" event={"ID":"702ac16e-58f7-402f-90e7-75d68204613a","Type":"ContainerDied","Data":"8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66"}
Oct 29 23:35:20.581961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.581636    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4" event={"ID":"702ac16e-58f7-402f-90e7-75d68204613a","Type":"ContainerDied","Data":"5aa6f5a4e383732c104e8f1a861f5aba10e4a1d327fde01afbd1cd629d4d20a8"}
Oct 29 23:35:20.581961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.581671    6203 scope.go:122] "RemoveContainer" containerID="8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66"
Oct 29 23:35:20.583541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.583514    6203 generic.go:356] "Generic (PLEG): container finished" podID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" containerID="48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11" exitCode=0
Oct 29 23:35:20.583674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.583557    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"
Oct 29 23:35:20.583674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.583602    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" event={"ID":"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873","Type":"ContainerDied","Data":"48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11"}
Oct 29 23:35:20.583674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.583663    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs" event={"ID":"a93ece91-3b4f-4cb2-a9df-1e1d4f51d873","Type":"ContainerDied","Data":"60816fadae7163980ac0a88f1cad09420a9b323c04c706373b1a79c0c72e226c"}
Oct 29 23:35:20.595939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.595917    6203 scope.go:122] "RemoveContainer" containerID="8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66"
Oct 29 23:35:20.596269 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:20.596248    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66\": container with ID starting with 8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66 not found: ID does not exist" containerID="8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66"
Oct 29 23:35:20.596333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.596276    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66"} err="failed to get container status \"8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66\": rpc error: code = NotFound desc = could not find container \"8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66\": container with ID starting with 8e0065b70e14b4bf3671260caf8b38108a24c48807fe010241859c6a0e2fbb66 not found: ID does not exist"
Oct 29 23:35:20.596333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.596294    6203 scope.go:122] "RemoveContainer" containerID="48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11"
Oct 29 23:35:20.608000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.607979    6203 scope.go:122] "RemoveContainer" containerID="48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11"
Oct 29 23:35:20.608351 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:20.608326    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11\": container with ID starting with 48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11 not found: ID does not exist" containerID="48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11"
Oct 29 23:35:20.608456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.608360    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11"} err="failed to get container status \"48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11\": rpc error: code = NotFound desc = could not find container \"48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11\": container with ID starting with 48ff39f66ae43299b1abeb395dd38af7a882228aaf6e177f8cdfc39491bcba11 not found: ID does not exist"
Oct 29 23:35:20.644986 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.644953    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"]
Oct 29 23:35:20.660353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.660321    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ae1364-7d9f6679db-8w7cs"]
Oct 29 23:35:20.694782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.694750    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"]
Oct 29 23:35:20.702027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:20.701998    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ae1364-78957d56c9-l68s4"]
Oct 29 23:35:21.882468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:21.882432    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"]
Oct 29 23:35:21.919328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:21.919291    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"]
Oct 29 23:35:22.018951 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.018919    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702ac16e-58f7-402f-90e7-75d68204613a" path="/var/lib/kubelet/pods/702ac16e-58f7-402f-90e7-75d68204613a/volumes"
Oct 29 23:35:22.019425 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.019405    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" path="/var/lib/kubelet/pods/a93ece91-3b4f-4cb2-a9df-1e1d4f51d873/volumes"
Oct 29 23:35:22.191002 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.190965    6203 patch_prober.go:29] interesting pod/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:35:22.191181 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.191028    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:35:22.845528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.845499    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:35:22.848703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.848674    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:35:22.979127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.979092    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-token\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-token\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.980927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.980872    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-podinfo\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.981163 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.981135    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-envoy\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.981381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.981357    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-socket\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.981606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.981581    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-socket\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.981764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.981750    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-certs\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.981939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.981862    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-credential-socket\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.982084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982071    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/150fb858-8a31-46dc-852c-46af8df85600-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150fb858-8a31-46dc-852c-46af8df85600-istio-podinfo\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.982188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982176    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-data\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.982284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982272    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-credential-socket\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.982389 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982376    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150fb858-8a31-46dc-852c-46af8df85600-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150fb858-8a31-46dc-852c-46af8df85600-istiod-ca-cert\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.982517 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982505    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-envoy\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.982626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982614    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-istio-token\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-istio-token\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.982777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982763    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-kube-api-access-tx8jg\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-kube-api-access-tx8jg\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.982930 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.982917    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944f6d7f-ecca-4b96-934b-ce03675d1770-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944f6d7f-ecca-4b96-934b-ce03675d1770-istiod-ca-cert\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.983064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.983049    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-certs\") pod \"150fb858-8a31-46dc-852c-46af8df85600\" (UID: \"150fb858-8a31-46dc-852c-46af8df85600\") "
Oct 29 23:35:22.983159 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.983147    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-kube-api-access-6vxb8\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-kube-api-access-6vxb8\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.983323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.983312    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-data\") pod \"944f6d7f-ecca-4b96-934b-ce03675d1770\" (UID: \"944f6d7f-ecca-4b96-934b-ce03675d1770\") "
Oct 29 23:35:22.984213 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.984193    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-data" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.986027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.986002    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-token" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:22.986365 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.986349    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-credential-socket" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.987250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.987221    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-credential-socket" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.987453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.987424    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-data" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.987709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.987652    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-socket" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.987941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.987922    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944f6d7f-ecca-4b96-934b-ce03675d1770-istiod-ca-cert" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:22.988295 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.988278    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-certs" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.988793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.988776    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-certs" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.991210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.991044    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-istio-token" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:22.991492 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.991417    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150fb858-8a31-46dc-852c-46af8df85600-istiod-ca-cert" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:22.992598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.992170    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-envoy" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.992598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.992559    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/150fb858-8a31-46dc-852c-46af8df85600-istio-podinfo" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:22.993246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.993215    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-envoy" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:22.993460 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.993396    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-podinfo" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:22.995189 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.995154    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-kube-api-access-6vxb8" pod "944f6d7f-ecca-4b96-934b-ce03675d1770" (UID: "944f6d7f-ecca-4b96-934b-ce03675d1770"). InnerVolumeSpecName "kube-api-access-6vxb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:22.997927 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:22.996809    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-socket" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:23.000873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.000840    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-kube-api-access-tx8jg" pod "150fb858-8a31-46dc-852c-46af8df85600" (UID: "150fb858-8a31-46dc-852c-46af8df85600"). InnerVolumeSpecName "kube-api-access-tx8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:23.084831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084745    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944f6d7f-ecca-4b96-934b-ce03675d1770-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.084831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084784    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.084831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084798    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vxb8\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-kube-api-access-6vxb8\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.084831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084814    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.084831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084829    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084842    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084854    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084868    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084894    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084908    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084922    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944f6d7f-ecca-4b96-934b-ce03675d1770-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084934    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150fb858-8a31-46dc-852c-46af8df85600-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084948    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084962    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084975    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150fb858-8a31-46dc-852c-46af8df85600-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.084988    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150fb858-8a31-46dc-852c-46af8df85600-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.085001    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.085273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.085014    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx8jg\" (UniqueName: \"kubernetes.io/projected/150fb858-8a31-46dc-852c-46af8df85600-kube-api-access-tx8jg\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.329580 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.329482    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:35:23.492344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492306    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-data\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492364    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-credential-socket\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492429    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-kube-api-access-d9k6d\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-kube-api-access-d9k6d\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492471    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-podinfo\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492503    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152dd6e0-c036-4a0b-a15f-a5cac127517e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152dd6e0-c036-4a0b-a15f-a5cac127517e-istiod-ca-cert\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492533    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-envoy\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492817 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492591    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-certs\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492817 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492652    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-token\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-token\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.492817 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492722    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-socket\") pod \"152dd6e0-c036-4a0b-a15f-a5cac127517e\" (UID: \"152dd6e0-c036-4a0b-a15f-a5cac127517e\") "
Oct 29 23:35:23.493242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492643    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-credential-socket" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:23.493242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492668    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-data" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:23.493242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492872    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-certs" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:23.493242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.492947    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152dd6e0-c036-4a0b-a15f-a5cac127517e-istiod-ca-cert" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:23.493514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.493296    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-socket" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:23.493568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.493517    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.493568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.493539    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/152dd6e0-c036-4a0b-a15f-a5cac127517e-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.493568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.493555    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.493726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.493571    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.493726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.493585    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.495321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.495292    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-kube-api-access-d9k6d" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "kube-api-access-d9k6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:23.495423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.495351    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-envoy" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:23.495593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.495558    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-podinfo" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:23.495979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.495956    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-token" pod "152dd6e0-c036-4a0b-a15f-a5cac127517e" (UID: "152dd6e0-c036-4a0b-a15f-a5cac127517e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:23.595583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.595542    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9k6d\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-kube-api-access-d9k6d\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.595583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.595585    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.595852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.595600    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.595852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.595615    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/152dd6e0-c036-4a0b-a15f-a5cac127517e-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.692416    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"]
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693162    6203 cpu_manager.go:414] "containerMap: removing container" podUID="702ac16e-58f7-402f-90e7-75d68204613a" containerName="payload-pre-processing"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693185    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702ac16e-58f7-402f-90e7-75d68204613a" containerName="payload-pre-processing"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693215    6203 cpu_manager.go:414] "containerMap: removing container" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693223    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693236    6203 cpu_manager.go:414] "containerMap: removing container" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693242    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693253    6203 cpu_manager.go:414] "containerMap: removing container" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693262    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693277    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" containerName="payload-processing"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693285    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" containerName="payload-processing"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693297    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693306    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693318    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" containerName="revoke-keys"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693326    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" containerName="revoke-keys"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693336    6203 cpu_manager.go:414] "containerMap: removing container" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693344    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693485    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" containerName="revoke-keys"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693500    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693516    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="702ac16e-58f7-402f-90e7-75d68204613a" containerName="payload-pre-processing"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693527    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d706de1-efaf-4c90-b710-eab26d3ed7f8" containerName="main"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693537    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="150fb858-8a31-46dc-852c-46af8df85600" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693544    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerName="istio-proxy"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693553    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93ece91-3b4f-4cb2-a9df-1e1d4f51d873" containerName="payload-processing"
Oct 29 23:35:23.708941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.693560    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93027eb-7194-4f70-b09c-d7a767d1715a" containerName="main"
Oct 29 23:35:23.710433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.710304    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.743472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.739293    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:35:23.743472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.740158    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:35:23.743472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.740260    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:35:23.781850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.770515    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"]
Oct 29 23:35:23.805820 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.805785    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.806203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.806185    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.806361 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.806347    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cklg2\" (UniqueName: \"kubernetes.io/projected/f52ddb5e-d6ce-4494-9281-c7940785627f-kube-api-access-cklg2\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.809776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.809746    6203 generic.go:356] "Generic (PLEG): container finished" podID="944f6d7f-ecca-4b96-934b-ce03675d1770" containerID="4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd" exitCode=0
Oct 29 23:35:23.810343 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.809996    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"
Oct 29 23:35:23.815758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.810016    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" event={"ID":"944f6d7f-ecca-4b96-934b-ce03675d1770","Type":"ContainerDied","Data":"4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd"}
Oct 29 23:35:23.815758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.813780    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m" event={"ID":"944f6d7f-ecca-4b96-934b-ce03675d1770","Type":"ContainerDied","Data":"40f71eb2d929d55b342075e882a1a41e4dbb39614a1fc8bb6f0d2c2a62429e8b"}
Oct 29 23:35:23.815758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.813809    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" event={"ID":"150fb858-8a31-46dc-852c-46af8df85600","Type":"ContainerDied","Data":"c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26"}
Oct 29 23:35:23.815758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.813154    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"
Oct 29 23:35:23.822928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.813086    6203 generic.go:356] "Generic (PLEG): container finished" podID="150fb858-8a31-46dc-852c-46af8df85600" containerID="c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26" exitCode=0
Oct 29 23:35:23.824233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.823062    6203 scope.go:122] "RemoveContainer" containerID="4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd"
Oct 29 23:35:23.826537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.824371    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds" event={"ID":"150fb858-8a31-46dc-852c-46af8df85600","Type":"ContainerDied","Data":"cb7b22da5e546fcf13b784d29ff905152c316f3e866c705f985ca65d60b1fc0d"}
Oct 29 23:35:23.828538 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.828487    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" event={"ID":"152dd6e0-c036-4a0b-a15f-a5cac127517e","Type":"ContainerDied","Data":"113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13"}
Oct 29 23:35:23.828925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.826316    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"
Oct 29 23:35:23.829984 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.826260    6203 generic.go:356] "Generic (PLEG): container finished" podID="152dd6e0-c036-4a0b-a15f-a5cac127517e" containerID="113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13" exitCode=0
Oct 29 23:35:23.830141 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.830122    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf" event={"ID":"152dd6e0-c036-4a0b-a15f-a5cac127517e","Type":"ContainerDied","Data":"a76944e46ff357042a71585964ff17a010e61c202a1a6f2fa5c92db7d4a60a47"}
Oct 29 23:35:23.830253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.828846    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:35:23.919943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.916997    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.919943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.917127    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.919943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.917164    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cklg2\" (UniqueName: \"kubernetes.io/projected/f52ddb5e-d6ce-4494-9281-c7940785627f-kube-api-access-cklg2\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.919943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.918195    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.958713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.958684    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.977904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.976662    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cklg2\" (UniqueName: \"kubernetes.io/projected/f52ddb5e-d6ce-4494-9281-c7940785627f-kube-api-access-cklg2\") pod \"maas-api-revoke-sub-1be84620-nkj5n\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:23.996072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.990187    6203 scope.go:122] "RemoveContainer" containerID="4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd"
Oct 29 23:35:23.999192 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:23.999004    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd\": container with ID starting with 4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd not found: ID does not exist" containerID="4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd"
Oct 29 23:35:23.999192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.999053    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd"} err="failed to get container status \"4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd\": rpc error: code = NotFound desc = could not find container \"4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd\": container with ID starting with 4431305ff7cb232bccfa4a4bce0107e2e9503386d61e11bfd3767fbce3eff9cd not found: ID does not exist"
Oct 29 23:35:23.999192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:23.999081    6203 scope.go:122] "RemoveContainer" containerID="c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26"
Oct 29 23:35:24.007794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.007767    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"]
Oct 29 23:35:24.031919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.029767    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-bx7ds"]
Oct 29 23:35:24.051146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.039366    6203 scope.go:122] "RemoveContainer" containerID="c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26"
Oct 29 23:35:24.051146 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:24.039711    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26\": container with ID starting with c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26 not found: ID does not exist" containerID="c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26"
Oct 29 23:35:24.051146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.039744    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26"} err="failed to get container status \"c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26\": rpc error: code = NotFound desc = could not find container \"c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26\": container with ID starting with c00b6ac5d4e0eb59f282c9110251a3bd74d767b23c1f465f7fb1819f1f60fb26 not found: ID does not exist"
Oct 29 23:35:24.051146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.039769    6203 scope.go:122] "RemoveContainer" containerID="113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13"
Oct 29 23:35:24.057040 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.056697    6203 scope.go:122] "RemoveContainer" containerID="113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13"
Oct 29 23:35:24.057213 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:24.057136    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13\": container with ID starting with 113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13 not found: ID does not exist" containerID="113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13"
Oct 29 23:35:24.057213 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.057175    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13"} err="failed to get container status \"113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13\": rpc error: code = NotFound desc = could not find container \"113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13\": container with ID starting with 113aa7e4397a3bd12d3d430a776d475f0796a05a46101e874ef05236ddf44e13 not found: ID does not exist"
Oct 29 23:35:24.097923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.093241    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"]
Oct 29 23:35:24.106924 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.104836    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-wtt9m"]
Oct 29 23:35:24.110307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.110284    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:24.211005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.210964    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"]
Oct 29 23:35:24.218289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.218259    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ae1364-openshift-default-5df5f95d69-xjgnf"]
Oct 29 23:35:24.424672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.424419    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"]
Oct 29 23:35:24.514916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.513854    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ae1784-77qqw"]
Oct 29 23:35:24.612043 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.611964    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"]
Oct 29 23:35:24.619318 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:35:24.619214    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52ddb5e_d6ce_4494_9281_c7940785627f.slice/crio-f66a27d86c7dd0e99eb81e14a460fb558f52a61d817c84cdb6c35220026a2649 WatchSource:0}: Error finding container f66a27d86c7dd0e99eb81e14a460fb558f52a61d817c84cdb6c35220026a2649: Status 404 returned error can't find the container with id f66a27d86c7dd0e99eb81e14a460fb558f52a61d817c84cdb6c35220026a2649
Oct 29 23:35:24.847168 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:24.847128    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n" event={"ID":"f52ddb5e-d6ce-4494-9281-c7940785627f","Type":"ContainerStarted","Data":"f66a27d86c7dd0e99eb81e14a460fb558f52a61d817c84cdb6c35220026a2649"}
Oct 29 23:35:25.861610 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:25.861573    6203 generic.go:356] "Generic (PLEG): container finished" podID="f52ddb5e-d6ce-4494-9281-c7940785627f" containerID="e459e4c0b21350ad2d9f4d32dc0ed8bdb571fbc079a08efb9244bd9fa113c809" exitCode=0
Oct 29 23:35:25.862011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:25.861640    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n" event={"ID":"f52ddb5e-d6ce-4494-9281-c7940785627f","Type":"ContainerDied","Data":"e459e4c0b21350ad2d9f4d32dc0ed8bdb571fbc079a08efb9244bd9fa113c809"}
Oct 29 23:35:26.018646 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:26.018610    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150fb858-8a31-46dc-852c-46af8df85600" path="/var/lib/kubelet/pods/150fb858-8a31-46dc-852c-46af8df85600/volumes"
Oct 29 23:35:26.019120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:26.019105    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152dd6e0-c036-4a0b-a15f-a5cac127517e" path="/var/lib/kubelet/pods/152dd6e0-c036-4a0b-a15f-a5cac127517e/volumes"
Oct 29 23:35:26.019488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:26.019477    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7336aa02-bb14-41d7-aa2d-9e69b19812a9" path="/var/lib/kubelet/pods/7336aa02-bb14-41d7-aa2d-9e69b19812a9/volumes"
Oct 29 23:35:26.020239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:26.020221    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944f6d7f-ecca-4b96-934b-ce03675d1770" path="/var/lib/kubelet/pods/944f6d7f-ecca-4b96-934b-ce03675d1770/volumes"
Oct 29 23:35:26.883277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:26.883229    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" podUID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" containerName="payload-processing" containerID="cri-o://594c67051364ba4aa37a53551541810c2f10a24487e33b61a1e2b8573085e88b" gracePeriod=25
Oct 29 23:35:26.920542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:26.920483    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" podUID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" containerName="payload-pre-processing" containerID="cri-o://ea24aa9b1d08811ec13822fdd7f976bcf6babb3535d1588a549d6c111ebeea2c" gracePeriod=25
Oct 29 23:35:27.034609 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.034583    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:27.071355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.071324    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-openshift-ca\") pod \"f52ddb5e-d6ce-4494-9281-c7940785627f\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") "
Oct 29 23:35:27.071504 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.071373    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-xks-ca\") pod \"f52ddb5e-d6ce-4494-9281-c7940785627f\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") "
Oct 29 23:35:27.071504 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.071414    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52ddb5e-d6ce-4494-9281-c7940785627f-kube-api-access-cklg2\" (UniqueName: \"kubernetes.io/projected/f52ddb5e-d6ce-4494-9281-c7940785627f-kube-api-access-cklg2\") pod \"f52ddb5e-d6ce-4494-9281-c7940785627f\" (UID: \"f52ddb5e-d6ce-4494-9281-c7940785627f\") "
Oct 29 23:35:27.071771 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.071738    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-openshift-ca" pod "f52ddb5e-d6ce-4494-9281-c7940785627f" (UID: "f52ddb5e-d6ce-4494-9281-c7940785627f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:27.071998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.071973    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:27.073566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.073544    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-xks-ca" pod "f52ddb5e-d6ce-4494-9281-c7940785627f" (UID: "f52ddb5e-d6ce-4494-9281-c7940785627f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:35:27.073566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.073557    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52ddb5e-d6ce-4494-9281-c7940785627f-kube-api-access-cklg2" pod "f52ddb5e-d6ce-4494-9281-c7940785627f" (UID: "f52ddb5e-d6ce-4494-9281-c7940785627f"). InnerVolumeSpecName "kube-api-access-cklg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:27.173589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.173555    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f52ddb5e-d6ce-4494-9281-c7940785627f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:27.173589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.173586    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cklg2\" (UniqueName: \"kubernetes.io/projected/f52ddb5e-d6ce-4494-9281-c7940785627f-kube-api-access-cklg2\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:27.883572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.883546    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"
Oct 29 23:35:27.883969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.883536    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n" event={"ID":"f52ddb5e-d6ce-4494-9281-c7940785627f","Type":"ContainerDied","Data":"f66a27d86c7dd0e99eb81e14a460fb558f52a61d817c84cdb6c35220026a2649"}
Oct 29 23:35:27.883969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:27.883650    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66a27d86c7dd0e99eb81e14a460fb558f52a61d817c84cdb6c35220026a2649"
Oct 29 23:35:28.795887 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.795855    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:35:28.892943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.892829    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ebd8bf-7fff-47e9-b706-6baf7f325180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebd8bf-7fff-47e9-b706-6baf7f325180-plugins-config-volume\") pod \"71ebd8bf-7fff-47e9-b706-6baf7f325180\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") "
Oct 29 23:35:28.893854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.892948    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ebd8bf-7fff-47e9-b706-6baf7f325180-kube-api-access-5dqqr\" (UniqueName: \"kubernetes.io/projected/71ebd8bf-7fff-47e9-b706-6baf7f325180-kube-api-access-5dqqr\") pod \"71ebd8bf-7fff-47e9-b706-6baf7f325180\" (UID: \"71ebd8bf-7fff-47e9-b706-6baf7f325180\") "
Oct 29 23:35:28.893854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.893302    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ebd8bf-7fff-47e9-b706-6baf7f325180-plugins-config-volume" pod "71ebd8bf-7fff-47e9-b706-6baf7f325180" (UID: "71ebd8bf-7fff-47e9-b706-6baf7f325180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:28.893854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.893500    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ebd8bf-7fff-47e9-b706-6baf7f325180-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:28.895203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.895173    6203 generic.go:356] "Generic (PLEG): container finished" podID="71ebd8bf-7fff-47e9-b706-6baf7f325180" containerID="e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3" exitCode=0
Oct 29 23:35:28.895386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.895248    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"
Oct 29 23:35:28.895386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.895253    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" event={"ID":"71ebd8bf-7fff-47e9-b706-6baf7f325180","Type":"ContainerDied","Data":"e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3"}
Oct 29 23:35:28.895386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.895359    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r" event={"ID":"71ebd8bf-7fff-47e9-b706-6baf7f325180","Type":"ContainerDied","Data":"15d55c27fcd0559e7ef0d454410df2e94c33be3b66d46def0746a6462766e474"}
Oct 29 23:35:28.895658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.895391    6203 scope.go:122] "RemoveContainer" containerID="e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3"
Oct 29 23:35:28.895918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.895871    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ebd8bf-7fff-47e9-b706-6baf7f325180-kube-api-access-5dqqr" pod "71ebd8bf-7fff-47e9-b706-6baf7f325180" (UID: "71ebd8bf-7fff-47e9-b706-6baf7f325180"). InnerVolumeSpecName "kube-api-access-5dqqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:28.911402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.911383    6203 scope.go:122] "RemoveContainer" containerID="e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3"
Oct 29 23:35:28.911738 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:28.911720    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3\": container with ID starting with e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3 not found: ID does not exist" containerID="e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3"
Oct 29 23:35:28.911803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.911752    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3"} err="failed to get container status \"e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3\": rpc error: code = NotFound desc = could not find container \"e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3\": container with ID starting with e60331179a8d6a9950f12acf7cc4a0d36d9e719a1e518435cb22fed6877b48f3 not found: ID does not exist"
Oct 29 23:35:28.994807 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:28.994770    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dqqr\" (UniqueName: \"kubernetes.io/projected/71ebd8bf-7fff-47e9-b706-6baf7f325180-kube-api-access-5dqqr\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:29.247638 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:29.247598    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"]
Oct 29 23:35:29.261518 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:29.261487    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ae1784-78546d6946-wb72r"]
Oct 29 23:35:30.018073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:30.018040    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ebd8bf-7fff-47e9-b706-6baf7f325180" path="/var/lib/kubelet/pods/71ebd8bf-7fff-47e9-b706-6baf7f325180/volumes"
Oct 29 23:35:32.293027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.292993    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"]
Oct 29 23:35:32.293477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.293238    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy" containerID="cri-o://a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935" gracePeriod=30
Oct 29 23:35:32.308017 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.307987    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"]
Oct 29 23:35:32.308221 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.308199    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" containerID="cri-o://9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4" gracePeriod=30
Oct 29 23:35:32.328113 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.328076    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"]
Oct 29 23:35:32.328449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.328417    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" containerID="cri-o://beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f" gracePeriod=30
Oct 29 23:35:32.426978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.424593    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:35:32.426978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:32.424672    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:35:33.043189 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:33.043148    6203 patch_prober.go:29] interesting pod/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:35:33.043367 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:33.043210    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:35:34.395484 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:34.395448    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" podUID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:34.395484 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:34.395468    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" podUID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:37.425116 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.425083    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"]
Oct 29 23:35:37.441467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.441415    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-aa7e96-c59pv"]
Oct 29 23:35:37.597975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.597945    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:35:37.638214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.638189    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:35:37.641101 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.641085    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:35:37.715651 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715614    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istiod-ca-cert\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.715850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715667    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-certs\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.715850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715712    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istiod-ca-cert\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.715850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715741    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-data\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.715850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715772    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-token\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-token\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715856    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-envoy\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715912    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-token\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-token\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715963    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-data\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.715995    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-certs\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716030    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-certs\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716070    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-podinfo\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716084    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-certs" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.716138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716108    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-envoy\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716153    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istiod-ca-cert" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716160    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istiod-ca-cert" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716165    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-credential-socket\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716221    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-socket\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716262    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29850449-f8b7-4c0f-9240-46fc86789ff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29850449-f8b7-4c0f-9240-46fc86789ff2-istiod-ca-cert\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716289    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-podinfo\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716396    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-credential-socket" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716436    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-data" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716443    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-socket\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716478    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-socket\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.716547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716515    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-envoy\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716597    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-kube-api-access-v9d2c\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-kube-api-access-v9d2c\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716611    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-data" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716673    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-credential-socket\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716694    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-certs" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716710    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-kube-api-access-jbwms\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-kube-api-access-jbwms\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716770    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-kube-api-access-mjx76\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-kube-api-access-mjx76\") pod \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\" (UID: \"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716802    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-podinfo\") pod \"29850449-f8b7-4c0f-9240-46fc86789ff2\" (UID: \"29850449-f8b7-4c0f-9240-46fc86789ff2\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716847    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-data\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716899    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-token\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-token\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716934    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-credential-socket\") pod \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\" (UID: \"c61cd131-3432-48e3-9b8f-d0a5990b34f4\") "
Oct 29 23:35:37.717136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.716958    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-certs" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717472    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717495    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717509    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717523    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717535    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717548    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717562    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.717697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717575    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.718150 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.717792    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-credential-socket" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.719489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.718762    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-socket" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.719489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.718915    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-kube-api-access-v9d2c" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "kube-api-access-v9d2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:37.719489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.718904    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-token" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:37.719489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.719033    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-envoy" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.719489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.719041    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-socket" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.719489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.719140    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-data" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.719489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.719251    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-socket" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.720219 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.720189    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-kube-api-access-mjx76" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "kube-api-access-mjx76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:37.720452 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.720429    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-credential-socket" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.720772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.720746    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29850449-f8b7-4c0f-9240-46fc86789ff2-istiod-ca-cert" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:37.721046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.721015    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-kube-api-access-jbwms" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "kube-api-access-jbwms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:37.721363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.721329    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-envoy" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.721568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.721536    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-podinfo" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:37.721568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.721540    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-token" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:37.721744 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.721567    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-token" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:37.722999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.722972    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-podinfo" pod "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" (UID: "d351e563-7bcd-4a78-b4f2-3e2ae3fcad10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:37.723119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.723065    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-envoy" pod "29850449-f8b7-4c0f-9240-46fc86789ff2" (UID: "29850449-f8b7-4c0f-9240-46fc86789ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:37.723384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.723350    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-podinfo" pod "c61cd131-3432-48e3-9b8f-d0a5990b34f4" (UID: "c61cd131-3432-48e3-9b8f-d0a5990b34f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:37.819208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819125    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9d2c\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-kube-api-access-v9d2c\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819155    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819166    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbwms\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-kube-api-access-jbwms\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819176    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjx76\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-kube-api-access-mjx76\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819186    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819200    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819210    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819219    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819227    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819236    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819243    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819250    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819257    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819265    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29850449-f8b7-4c0f-9240-46fc86789ff2-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819277    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29850449-f8b7-4c0f-9240-46fc86789ff2-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819288    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819295    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819307    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:37.819529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:37.819320    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c61cd131-3432-48e3-9b8f-d0a5990b34f4-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:38.018015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.017977    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50efe9a-cfd5-490d-aed9-1fe2d4cf362f" path="/var/lib/kubelet/pods/b50efe9a-cfd5-490d-aed9-1fe2d4cf362f/volumes"
Oct 29 23:35:38.022272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.022244    6203 generic.go:356] "Generic (PLEG): container finished" podID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerID="9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4" exitCode=0
Oct 29 23:35:38.022472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.022302    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"
Oct 29 23:35:38.022472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.022327    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" event={"ID":"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10","Type":"ContainerDied","Data":"9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4"}
Oct 29 23:35:38.022472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.022390    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7" event={"ID":"d351e563-7bcd-4a78-b4f2-3e2ae3fcad10","Type":"ContainerDied","Data":"b2eeca9a80d7f955be0107717026a608e69493989cb2f4a9140214ee3fda6e92"}
Oct 29 23:35:38.022472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.022420    6203 scope.go:122] "RemoveContainer" containerID="9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4"
Oct 29 23:35:38.024219 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.024193    6203 generic.go:356] "Generic (PLEG): container finished" podID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerID="a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935" exitCode=0
Oct 29 23:35:38.024332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.024242    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"
Oct 29 23:35:38.024332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.024268    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" event={"ID":"29850449-f8b7-4c0f-9240-46fc86789ff2","Type":"ContainerDied","Data":"a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935"}
Oct 29 23:35:38.024449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.024378    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z" event={"ID":"29850449-f8b7-4c0f-9240-46fc86789ff2","Type":"ContainerDied","Data":"f17d2c56ba1d999848a4dc229ddf273889fce44afcef4863c4e6ec49b5238631"}
Oct 29 23:35:38.026922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.026898    6203 generic.go:356] "Generic (PLEG): container finished" podID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerID="beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f" exitCode=0
Oct 29 23:35:38.027020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.026930    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" event={"ID":"c61cd131-3432-48e3-9b8f-d0a5990b34f4","Type":"ContainerDied","Data":"beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f"}
Oct 29 23:35:38.027020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.026946    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"
Oct 29 23:35:38.027020 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.026964    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6" event={"ID":"c61cd131-3432-48e3-9b8f-d0a5990b34f4","Type":"ContainerDied","Data":"8d614f0d8cf6cf818f809207fa251eece20633113fc770c40c5f2e89c0e7004b"}
Oct 29 23:35:38.035670 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.035652    6203 scope.go:122] "RemoveContainer" containerID="9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4"
Oct 29 23:35:38.035993 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:38.035976    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4\": container with ID starting with 9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4 not found: ID does not exist" containerID="9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4"
Oct 29 23:35:38.036073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.036002    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4"} err="failed to get container status \"9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4\": rpc error: code = NotFound desc = could not find container \"9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4\": container with ID starting with 9d54a6974409e3d2f64bcc6293225103c4cb54f3aaf0f17be5aedc2d23ca23b4 not found: ID does not exist"
Oct 29 23:35:38.036073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.036020    6203 scope.go:122] "RemoveContainer" containerID="a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935"
Oct 29 23:35:38.046500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.046482    6203 scope.go:122] "RemoveContainer" containerID="a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935"
Oct 29 23:35:38.046735 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:38.046718    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935\": container with ID starting with a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935 not found: ID does not exist" containerID="a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935"
Oct 29 23:35:38.046791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.046742    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935"} err="failed to get container status \"a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935\": rpc error: code = NotFound desc = could not find container \"a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935\": container with ID starting with a6238d6e40d6200862bb4902e8f87f00c375cb4d8832715a54265b51f6577935 not found: ID does not exist"
Oct 29 23:35:38.046791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.046758    6203 scope.go:122] "RemoveContainer" containerID="beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f"
Oct 29 23:35:38.055923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.055907    6203 scope.go:122] "RemoveContainer" containerID="beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f"
Oct 29 23:35:38.056194 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:38.056177    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f\": container with ID starting with beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f not found: ID does not exist" containerID="beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f"
Oct 29 23:35:38.056256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.056199    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f"} err="failed to get container status \"beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f\": rpc error: code = NotFound desc = could not find container \"beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f\": container with ID starting with beb317963005f3536f6c9af62392761992ff2b7af132a8129d655665a6e5fb6f not found: ID does not exist"
Oct 29 23:35:38.120403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.120321    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"]
Oct 29 23:35:38.133648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.133615    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-h82t7"]
Oct 29 23:35:38.162847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.162818    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"]
Oct 29 23:35:38.166971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.166948    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-frw4z"]
Oct 29 23:35:38.206741 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.206712    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"]
Oct 29 23:35:38.210231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:38.210207    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ae1784-openshift-default-5b66bf84bf-mv9v6"]
Oct 29 23:35:39.959603 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:39.959559    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:35:40.019018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:40.018984    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" path="/var/lib/kubelet/pods/29850449-f8b7-4c0f-9240-46fc86789ff2/volumes"
Oct 29 23:35:40.019453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:40.019441    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" path="/var/lib/kubelet/pods/c61cd131-3432-48e3-9b8f-d0a5990b34f4/volumes"
Oct 29 23:35:40.019827 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:40.019815    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" path="/var/lib/kubelet/pods/d351e563-7bcd-4a78-b4f2-3e2ae3fcad10/volumes"
Oct 29 23:35:44.396000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.395962    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" podUID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:44.396000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.395979    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" podUID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:35:44.455641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.455607    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"]
Oct 29 23:35:44.455901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.455842    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy" containerID="cri-o://bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744" gracePeriod=30
Oct 29 23:35:44.472528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.472493    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"]
Oct 29 23:35:44.472714 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.472694    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" containerID="cri-o://0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42" gracePeriod=30
Oct 29 23:35:44.477211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.477186    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"]
Oct 29 23:35:44.477389 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:44.477369    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy" containerID="cri-o://a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d" gracePeriod=30
Oct 29 23:35:46.603291 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:46.603260    6203 scope.go:122] "RemoveContainer" containerID="9630a040c2492228b0ea7a9609742fb4a717b9f98ddbde8085e2b352a6074e4e"
Oct 29 23:35:46.613089 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:46.613063    6203 scope.go:122] "RemoveContainer" containerID="f7acdfd490c92430d58f57afaa94322858d17c0a2c7900d682da19586583ba6a"
Oct 29 23:35:46.623570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:46.623550    6203 scope.go:122] "RemoveContainer" containerID="9a759885cb698474ac63ef1379da39eb80fd357794d3f31f40dbbeabe9a0db3d"
Oct 29 23:35:46.635317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:46.635294    6203 scope.go:122] "RemoveContainer" containerID="084e0be52fe399878629458d4fd18366e94a3f8ca7c1ca2cf2eceaa67ae7d19a"
Oct 29 23:35:46.643789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:46.643768    6203 scope.go:122] "RemoveContainer" containerID="dcacb44081dc9ad10bc523063cff32c72b0bc5b14a8a32bab3c961afa303435c"
Oct 29 23:35:46.652254 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:46.652235    6203 scope.go:122] "RemoveContainer" containerID="6326991daf5f2b83cd412606f6d7cf1ca18f092f0c898645406245c7d3b0fb3d"
Oct 29 23:35:46.660963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:46.660946    6203 scope.go:122] "RemoveContainer" containerID="909d3c5da8d40e7c34bc797063172c93c76723c9e133c6dfbbe00518f13af1e4"
Oct 29 23:35:47.110399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.110355    6203 generic.go:356] "Generic (PLEG): container finished" podID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" containerID="594c67051364ba4aa37a53551541810c2f10a24487e33b61a1e2b8573085e88b" exitCode=0
Oct 29 23:35:47.110574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.110540    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" event={"ID":"ee361d0d-7e9f-4d21-899b-0e360e7067ac","Type":"ContainerDied","Data":"594c67051364ba4aa37a53551541810c2f10a24487e33b61a1e2b8573085e88b"}
Oct 29 23:35:47.113050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.112856    6203 generic.go:356] "Generic (PLEG): container finished" podID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" containerID="ea24aa9b1d08811ec13822fdd7f976bcf6babb3535d1588a549d6c111ebeea2c" exitCode=0
Oct 29 23:35:47.113050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.112916    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" event={"ID":"3d39d18a-c0f4-4856-a880-e9bee6cdb224","Type":"ContainerDied","Data":"ea24aa9b1d08811ec13822fdd7f976bcf6babb3535d1588a549d6c111ebeea2c"}
Oct 29 23:35:47.173709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.173683    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:35:47.186431 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.186410    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:35:47.226399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.226372    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee361d0d-7e9f-4d21-899b-0e360e7067ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee361d0d-7e9f-4d21-899b-0e360e7067ac-plugins-config-volume\") pod \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") "
Oct 29 23:35:47.226544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.226447    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d39d18a-c0f4-4856-a880-e9bee6cdb224-kube-api-access-78tmx\" (UniqueName: \"kubernetes.io/projected/3d39d18a-c0f4-4856-a880-e9bee6cdb224-kube-api-access-78tmx\") pod \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") "
Oct 29 23:35:47.226544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.226492    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee361d0d-7e9f-4d21-899b-0e360e7067ac-kube-api-access-gdd6c\" (UniqueName: \"kubernetes.io/projected/ee361d0d-7e9f-4d21-899b-0e360e7067ac-kube-api-access-gdd6c\") pod \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\" (UID: \"ee361d0d-7e9f-4d21-899b-0e360e7067ac\") "
Oct 29 23:35:47.226544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.226532    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d39d18a-c0f4-4856-a880-e9bee6cdb224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d39d18a-c0f4-4856-a880-e9bee6cdb224-plugins-config-volume\") pod \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\" (UID: \"3d39d18a-c0f4-4856-a880-e9bee6cdb224\") "
Oct 29 23:35:47.226823 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.226795    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee361d0d-7e9f-4d21-899b-0e360e7067ac-plugins-config-volume" pod "ee361d0d-7e9f-4d21-899b-0e360e7067ac" (UID: "ee361d0d-7e9f-4d21-899b-0e360e7067ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:47.226999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.226974    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d39d18a-c0f4-4856-a880-e9bee6cdb224-plugins-config-volume" pod "3d39d18a-c0f4-4856-a880-e9bee6cdb224" (UID: "3d39d18a-c0f4-4856-a880-e9bee6cdb224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:47.228831 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.228808    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee361d0d-7e9f-4d21-899b-0e360e7067ac-kube-api-access-gdd6c" pod "ee361d0d-7e9f-4d21-899b-0e360e7067ac" (UID: "ee361d0d-7e9f-4d21-899b-0e360e7067ac"). InnerVolumeSpecName "kube-api-access-gdd6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:47.228969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.228871    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d39d18a-c0f4-4856-a880-e9bee6cdb224-kube-api-access-78tmx" pod "3d39d18a-c0f4-4856-a880-e9bee6cdb224" (UID: "3d39d18a-c0f4-4856-a880-e9bee6cdb224"). InnerVolumeSpecName "kube-api-access-78tmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:47.327548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.327510    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee361d0d-7e9f-4d21-899b-0e360e7067ac-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:47.327548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.327541    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78tmx\" (UniqueName: \"kubernetes.io/projected/3d39d18a-c0f4-4856-a880-e9bee6cdb224-kube-api-access-78tmx\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:47.327548 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.327553    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdd6c\" (UniqueName: \"kubernetes.io/projected/ee361d0d-7e9f-4d21-899b-0e360e7067ac-kube-api-access-gdd6c\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:47.328114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.327562    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d39d18a-c0f4-4856-a880-e9bee6cdb224-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:47.803328 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.803284    6203 patch_prober.go:29] interesting pod/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:35:47.803708 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:47.803346    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:35:48.121127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.121047    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"
Oct 29 23:35:48.121127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.121084    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz" event={"ID":"ee361d0d-7e9f-4d21-899b-0e360e7067ac","Type":"ContainerDied","Data":"983fed6bed7d47f0c5427ed8fdcba10712d7afdf383ea307e9f0d9925a118989"}
Oct 29 23:35:48.121355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.121136    6203 scope.go:122] "RemoveContainer" containerID="594c67051364ba4aa37a53551541810c2f10a24487e33b61a1e2b8573085e88b"
Oct 29 23:35:48.122853 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.122830    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"
Oct 29 23:35:48.122971 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.122864    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q" event={"ID":"3d39d18a-c0f4-4856-a880-e9bee6cdb224","Type":"ContainerDied","Data":"0922f16bb892abac2fb86235f36338b90b505ee6a467ce4dda791e2b8333bcb8"}
Oct 29 23:35:48.131214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.131186    6203 scope.go:122] "RemoveContainer" containerID="ea24aa9b1d08811ec13822fdd7f976bcf6babb3535d1588a549d6c111ebeea2c"
Oct 29 23:35:48.195689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.195659    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"]
Oct 29 23:35:48.209995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.209959    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-aa7e96-69dcf5cc99-wsj5q"]
Oct 29 23:35:48.240278 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.240249    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"]
Oct 29 23:35:48.244402 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:48.244381    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-aa7e96-b7f9fb496-97lbz"]
Oct 29 23:35:49.737000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.736916    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:35:49.781941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.781914    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:35:49.785018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.784997    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:35:49.854011 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.853979    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a7f61d-89b9-4358-84fa-75e6d6201317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7f61d-89b9-4358-84fa-75e6d6201317-istiod-ca-cert\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.854202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854017    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-socket\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.854202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854053    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-envoy\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.854202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854087    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-certs\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.854202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854116    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eeca893-2cf6-463a-b808-e49d9efa768d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eeca893-2cf6-463a-b808-e49d9efa768d-istiod-ca-cert\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.854202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854148    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-token\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-token\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.854202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854177    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-certs\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.854532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854248    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-token\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-token\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.854532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854387    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-socket" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.854749 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.854719    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eeca893-2cf6-463a-b808-e49d9efa768d-istiod-ca-cert" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:49.855043 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855016    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-certs" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.855240 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855220    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-certs" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.855345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855325    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-credential-socket\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.855414 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855372    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istiod-ca-cert\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.855414 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855402    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-data\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.855526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855429    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-podinfo\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.855526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855462    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-kube-api-access-c2k59\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-kube-api-access-c2k59\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.855526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855494    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-certs\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.855526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855523    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-envoy\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.855776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855552    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-socket\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.855776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855564    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a7f61d-89b9-4358-84fa-75e6d6201317-istiod-ca-cert" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:49.855776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855605    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-kube-api-access-wvnpp\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-kube-api-access-wvnpp\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.855776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855645    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-data\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.855776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855676    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-token\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-token\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.855776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855729    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-socket\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.856121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855787    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-data\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.856121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.855822    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-podinfo\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.856121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.856042    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-socket" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.856121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.856108    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-socket" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.856335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.856289    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-data" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.856911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.856861    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-credential-socket\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.857006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.856954    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-podinfo\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.857006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.856990    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-envoy" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.857121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.856985    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-envoy\") pod \"7eeca893-2cf6-463a-b808-e49d9efa768d\" (UID: \"7eeca893-2cf6-463a-b808-e49d9efa768d\") "
Oct 29 23:35:49.857121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.857100    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-credential-socket\") pod \"05a7f61d-89b9-4358-84fa-75e6d6201317\" (UID: \"05a7f61d-89b9-4358-84fa-75e6d6201317\") "
Oct 29 23:35:49.857220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.857137    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-kube-api-access-ltqnb\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-kube-api-access-ltqnb\") pod \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\" (UID: \"7e123f70-499b-4e8d-a0ea-3655e76e94a1\") "
Oct 29 23:35:49.857408 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.857381    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-token" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:49.857613 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.857593    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-credential-socket" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.857687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.857617    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-data" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.857787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.857393    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-credential-socket" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.857907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.857410    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-certs" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.858046 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858028    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.858155 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858142    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.858250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858238    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a7f61d-89b9-4358-84fa-75e6d6201317-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.858439 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858424    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.858549 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858535    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.858644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858633    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.858729 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858717    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eeca893-2cf6-463a-b808-e49d9efa768d-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.858853 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.858842    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.859058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.859044    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.859174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.859156    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.859292 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.859272    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-token" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:49.859690 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.859666    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-podinfo" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:49.859690 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.859671    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istiod-ca-cert" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:49.859904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.859856    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-data" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.860069 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.859910    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-credential-socket" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.861143 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.861107    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-podinfo" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:49.861241 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.861190    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-kube-api-access-ltqnb" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "kube-api-access-ltqnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:49.861323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.861298    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-envoy" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.862021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.861995    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-podinfo" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:49.862356 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.862331    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-kube-api-access-wvnpp" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "kube-api-access-wvnpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:49.862722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.862704    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-token" pod "05a7f61d-89b9-4358-84fa-75e6d6201317" (UID: "05a7f61d-89b9-4358-84fa-75e6d6201317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:49.864728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.864700    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-kube-api-access-c2k59" pod "7eeca893-2cf6-463a-b808-e49d9efa768d" (UID: "7eeca893-2cf6-463a-b808-e49d9efa768d"). InnerVolumeSpecName "kube-api-access-c2k59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:49.865281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.865255    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-envoy" pod "7e123f70-499b-4e8d-a0ea-3655e76e94a1" (UID: "7e123f70-499b-4e8d-a0ea-3655e76e94a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:49.960393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960355    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvnpp\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-kube-api-access-wvnpp\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960390    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960401    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960410    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960418    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960426    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960436    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960444    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960452    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltqnb\" (UniqueName: \"kubernetes.io/projected/7e123f70-499b-4e8d-a0ea-3655e76e94a1-kube-api-access-ltqnb\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960460    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960469    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eeca893-2cf6-463a-b808-e49d9efa768d-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960478    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960486    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960493    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960501    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2k59\" (UniqueName: \"kubernetes.io/projected/7eeca893-2cf6-463a-b808-e49d9efa768d-kube-api-access-c2k59\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960509    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a7f61d-89b9-4358-84fa-75e6d6201317-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:49.960644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:49.960517    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e123f70-499b-4e8d-a0ea-3655e76e94a1-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:50.018990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.018910    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" path="/var/lib/kubelet/pods/3d39d18a-c0f4-4856-a880-e9bee6cdb224/volumes"
Oct 29 23:35:50.019309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.019296    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" path="/var/lib/kubelet/pods/ee361d0d-7e9f-4d21-899b-0e360e7067ac/volumes"
Oct 29 23:35:50.142301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.142259    6203 generic.go:356] "Generic (PLEG): container finished" podID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerID="a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d" exitCode=0
Oct 29 23:35:50.142505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.142315    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"
Oct 29 23:35:50.142505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.142348    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" event={"ID":"7e123f70-499b-4e8d-a0ea-3655e76e94a1","Type":"ContainerDied","Data":"a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d"}
Oct 29 23:35:50.142505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.142416    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm" event={"ID":"7e123f70-499b-4e8d-a0ea-3655e76e94a1","Type":"ContainerDied","Data":"abc1ed5827436bca69dc885e80c187d6813943057f77ca8d722a8d95ad7e87b3"}
Oct 29 23:35:50.142505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.142445    6203 scope.go:122] "RemoveContainer" containerID="a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d"
Oct 29 23:35:50.144433 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.144407    6203 generic.go:356] "Generic (PLEG): container finished" podID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerID="bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744" exitCode=0
Oct 29 23:35:50.144547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.144487    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"
Oct 29 23:35:50.144547 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.144497    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" event={"ID":"7eeca893-2cf6-463a-b808-e49d9efa768d","Type":"ContainerDied","Data":"bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744"}
Oct 29 23:35:50.144654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.144544    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn" event={"ID":"7eeca893-2cf6-463a-b808-e49d9efa768d","Type":"ContainerDied","Data":"38b666d2a49c452c3d2eb3a5d5ea0265537397f2712017d55ca750faae3ccab0"}
Oct 29 23:35:50.146276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.146257    6203 generic.go:356] "Generic (PLEG): container finished" podID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerID="0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42" exitCode=0
Oct 29 23:35:50.146378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.146283    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" event={"ID":"05a7f61d-89b9-4358-84fa-75e6d6201317","Type":"ContainerDied","Data":"0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42"}
Oct 29 23:35:50.146378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.146301    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"
Oct 29 23:35:50.146378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.146313    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8" event={"ID":"05a7f61d-89b9-4358-84fa-75e6d6201317","Type":"ContainerDied","Data":"6394e606efdfbaa7e95eb0ee3ea6f4812663de46a73c296f1e7ee517e5a8a7d1"}
Oct 29 23:35:50.153846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.153823    6203 scope.go:122] "RemoveContainer" containerID="a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d"
Oct 29 23:35:50.154197 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:50.154177    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d\": container with ID starting with a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d not found: ID does not exist" containerID="a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d"
Oct 29 23:35:50.154246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.154207    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d"} err="failed to get container status \"a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d\": rpc error: code = NotFound desc = could not find container \"a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d\": container with ID starting with a4fbfcad740a0f4ac58d2a358e1dd0949168f828d44389ec0805f8e6bd0e783d not found: ID does not exist"
Oct 29 23:35:50.154246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.154225    6203 scope.go:122] "RemoveContainer" containerID="bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744"
Oct 29 23:35:50.163632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.163615    6203 scope.go:122] "RemoveContainer" containerID="05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b"
Oct 29 23:35:50.172727 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.172708    6203 scope.go:122] "RemoveContainer" containerID="bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744"
Oct 29 23:35:50.173032 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:50.173012    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744\": container with ID starting with bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744 not found: ID does not exist" containerID="bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744"
Oct 29 23:35:50.173125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.173038    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744"} err="failed to get container status \"bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744\": rpc error: code = NotFound desc = could not find container \"bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744\": container with ID starting with bab843ad59e89bc72a213bfd4e5fb0f839cbfad4f2309c748d76c148dc973744 not found: ID does not exist"
Oct 29 23:35:50.173125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.173056    6203 scope.go:122] "RemoveContainer" containerID="05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b"
Oct 29 23:35:50.173313 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:50.173292    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b\": container with ID starting with 05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b not found: ID does not exist" containerID="05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b"
Oct 29 23:35:50.173377 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.173320    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b"} err="failed to get container status \"05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b\": rpc error: code = NotFound desc = could not find container \"05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b\": container with ID starting with 05a27a80b74db17e52b6da415e03ec25c645c6269e54a2c33dc6e691121ca93b not found: ID does not exist"
Oct 29 23:35:50.173377 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.173337    6203 scope.go:122] "RemoveContainer" containerID="0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42"
Oct 29 23:35:50.183188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.183162    6203 scope.go:122] "RemoveContainer" containerID="0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42"
Oct 29 23:35:50.183456 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:50.183438    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42\": container with ID starting with 0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42 not found: ID does not exist" containerID="0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42"
Oct 29 23:35:50.183520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.183462    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42"} err="failed to get container status \"0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42\": rpc error: code = NotFound desc = could not find container \"0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42\": container with ID starting with 0d1b6a696cccf7f3ac68cc44273dcfd0d75451d3c0602b19b7efe67976b5ad42 not found: ID does not exist"
Oct 29 23:35:50.203542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.203513    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"]
Oct 29 23:35:50.217893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.217849    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-6xpq8"]
Oct 29 23:35:50.262291 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.262258    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"]
Oct 29 23:35:50.273081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.273007    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-w9tzm"]
Oct 29 23:35:50.323529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.323495    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"]
Oct 29 23:35:50.328515 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:50.328483    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-aa7e96-openshift-default-7dd86477ff-gfhhn"]
Oct 29 23:35:52.073903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:52.064645    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" path="/var/lib/kubelet/pods/05a7f61d-89b9-4358-84fa-75e6d6201317/volumes"
Oct 29 23:35:52.073903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:52.065341    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" path="/var/lib/kubelet/pods/7e123f70-499b-4e8d-a0ea-3655e76e94a1/volumes"
Oct 29 23:35:52.074750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:52.074699    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" path="/var/lib/kubelet/pods/7eeca893-2cf6-463a-b808-e49d9efa768d/volumes"
Oct 29 23:35:53.790175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:53.790142    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"]
Oct 29 23:35:53.790626 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:53.790399    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy" containerID="cri-o://c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2" gracePeriod=30
Oct 29 23:35:53.804976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:53.804944    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"]
Oct 29 23:35:53.805239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:53.805192    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy" containerID="cri-o://06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376" gracePeriod=30
Oct 29 23:35:59.045372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.045345    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:35:59.088687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.088662    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:35:59.178655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178623    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-data\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.178655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178661    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-token\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178684    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-podinfo\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178707    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istiod-ca-cert\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178727    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-kube-api-access-6zwhw\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-kube-api-access-6zwhw\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178764    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istiod-ca-cert\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178817    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-envoy\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178845    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-socket\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178907    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-certs\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.178936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178928    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-credential-socket\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178958    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-podinfo\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.178984    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-data" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179026    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-token\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-token\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179059    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-socket\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179086    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-credential-socket\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179138    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-data\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179193    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-envoy\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179196    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-certs" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179207    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istiod-ca-cert" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179278    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-kube-api-access-bcptn\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-kube-api-access-bcptn\") pod \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\" (UID: \"7fa18428-1dc5-4b21-87b1-f5860cc77eeb\") "
Oct 29 23:35:59.179352 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179314    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-certs\") pod \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\" (UID: \"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f\") "
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179459    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-socket" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179521    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istiod-ca-cert" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179640    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-certs" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179778    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179796    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179810    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179824    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179836    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.179955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.179849    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.180426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.180101    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-credential-socket" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.181530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.181503    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-credential-socket" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.181663 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.181562    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-socket" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.181663 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.181621    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-data" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.181663 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.181653    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-envoy" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.181860 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.181689    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-token" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:59.182370 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.182343    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-token" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:59.182498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.182394    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-podinfo" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:59.182498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.182395    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-envoy" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:35:59.182643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.182627    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-podinfo" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:35:59.182727 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.182712    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-kube-api-access-6zwhw" pod "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" (UID: "2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f"). InnerVolumeSpecName "kube-api-access-6zwhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:59.183133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.183106    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-kube-api-access-bcptn" pod "7fa18428-1dc5-4b21-87b1-f5860cc77eeb" (UID: "7fa18428-1dc5-4b21-87b1-f5860cc77eeb"). InnerVolumeSpecName "kube-api-access-bcptn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:35:59.235436 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.235402    6203 generic.go:356] "Generic (PLEG): container finished" podID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerID="c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2" exitCode=0
Oct 29 23:35:59.235593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.235453    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"
Oct 29 23:35:59.235593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.235488    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" event={"ID":"7fa18428-1dc5-4b21-87b1-f5860cc77eeb","Type":"ContainerDied","Data":"c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2"}
Oct 29 23:35:59.235593 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.235585    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg" event={"ID":"7fa18428-1dc5-4b21-87b1-f5860cc77eeb","Type":"ContainerDied","Data":"f9498fb3fce0a64a8957751f8eb1cacc644f0c750af7ddbeb2409db18ca3bf3a"}
Oct 29 23:35:59.235759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.235615    6203 scope.go:122] "RemoveContainer" containerID="c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2"
Oct 29 23:35:59.236996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.236974    6203 generic.go:356] "Generic (PLEG): container finished" podID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerID="06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376" exitCode=0
Oct 29 23:35:59.237106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.237009    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"
Oct 29 23:35:59.237106 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.237055    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" event={"ID":"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f","Type":"ContainerDied","Data":"06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376"}
Oct 29 23:35:59.237220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.237111    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx" event={"ID":"2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f","Type":"ContainerDied","Data":"d8a2bac390ebc0b64d9a724404e1f2fbb11b14ea5c480bdc713a63c845125d61"}
Oct 29 23:35:59.246777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.246755    6203 scope.go:122] "RemoveContainer" containerID="c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2"
Oct 29 23:35:59.247264 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:59.247241    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2\": container with ID starting with c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2 not found: ID does not exist" containerID="c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2"
Oct 29 23:35:59.247337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.247271    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2"} err="failed to get container status \"c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2\": rpc error: code = NotFound desc = could not find container \"c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2\": container with ID starting with c398ca7286bb3cfa9d6392bdc534f39379fa0056317cc7c7dc2a224eba4596c2 not found: ID does not exist"
Oct 29 23:35:59.247337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.247288    6203 scope.go:122] "RemoveContainer" containerID="06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376"
Oct 29 23:35:59.256350 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.256333    6203 scope.go:122] "RemoveContainer" containerID="06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376"
Oct 29 23:35:59.256624 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:35:59.256602    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376\": container with ID starting with 06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376 not found: ID does not exist" containerID="06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376"
Oct 29 23:35:59.256697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.256636    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376"} err="failed to get container status \"06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376\": rpc error: code = NotFound desc = could not find container \"06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376\": container with ID starting with 06fc4fd56ef8963b813baec6ae88a462fc088aa541ae4915311185686a92c376 not found: ID does not exist"
Oct 29 23:35:59.280672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280645    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zwhw\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-kube-api-access-6zwhw\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280669    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280682    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280693    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280726    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280736    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280745    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280755    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280763    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280774    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcptn\" (UniqueName: \"kubernetes.io/projected/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-kube-api-access-bcptn\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280785    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.280837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.280792    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa18428-1dc5-4b21-87b1-f5860cc77eeb-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:35:59.303466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.303434    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"]
Oct 29 23:35:59.312805 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.312781    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-hjjpg"]
Oct 29 23:35:59.352058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.351960    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"]
Oct 29 23:35:59.355717 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:35:59.355697    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-qpzfx"]
Oct 29 23:36:00.017584 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:00.017553    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" path="/var/lib/kubelet/pods/2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f/volumes"
Oct 29 23:36:00.018022 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:00.018007    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" path="/var/lib/kubelet/pods/7fa18428-1dc5-4b21-87b1-f5860cc77eeb/volumes"
Oct 29 23:36:08.887149 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887113    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"]
Oct 29 23:36:08.887707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887685    6203 cpu_manager.go:414] "containerMap: removing container" podUID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" containerName="payload-pre-processing"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887710    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" containerName="payload-pre-processing"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887728    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887735    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887748    6203 cpu_manager.go:414] "containerMap: removing container" podUID="71ebd8bf-7fff-47e9-b706-6baf7f325180" containerName="payload-processing"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887755    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ebd8bf-7fff-47e9-b706-6baf7f325180" containerName="payload-processing"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887772    6203 cpu_manager.go:414] "containerMap: removing container" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887779    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887793    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy"
Oct 29 23:36:08.887800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887801    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887816    6203 cpu_manager.go:414] "containerMap: removing container" podUID="f52ddb5e-d6ce-4494-9281-c7940785627f" containerName="revoke-keys"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887827    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52ddb5e-d6ce-4494-9281-c7940785627f" containerName="revoke-keys"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887850    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887858    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887891    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887900    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887918    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887925    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887936    6203 cpu_manager.go:414] "containerMap: removing container" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887943    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887959    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887966    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887976    6203 cpu_manager.go:414] "containerMap: removing container" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887985    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.887996    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" containerName="payload-processing"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888003    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" containerName="payload-processing"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888126    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d39d18a-c0f4-4856-a880-e9bee6cdb224" containerName="payload-pre-processing"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888142    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa18428-1dc5-4b21-87b1-f5860cc77eeb" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888151    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ebd8bf-7fff-47e9-b706-6baf7f325180" containerName="payload-processing"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888165    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="d351e563-7bcd-4a78-b4f2-3e2ae3fcad10" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888176    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1db0e3-77bd-47b0-aa83-3fc0cb08f77f" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888187    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52ddb5e-d6ce-4494-9281-c7940785627f" containerName="revoke-keys"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888200    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61cd131-3432-48e3-9b8f-d0a5990b34f4" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888212    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e123f70-499b-4e8d-a0ea-3655e76e94a1" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888223    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888233    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a7f61d-89b9-4358-84fa-75e6d6201317" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888241    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eeca893-2cf6-463a-b808-e49d9efa768d" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888249    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="29850449-f8b7-4c0f-9240-46fc86789ff2" containerName="istio-proxy"
Oct 29 23:36:08.888393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.888258    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee361d0d-7e9f-4d21-899b-0e360e7067ac" containerName="payload-processing"
Oct 29 23:36:08.891799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.891779    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.930923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.930861    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"]
Oct 29 23:36:08.973493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973454    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.973493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973492    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90102371-81a6-4419-a23e-72f0ba01c595-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.973721 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973612    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90102371-81a6-4419-a23e-72f0ba01c595-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.973765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973716    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6869\" (UniqueName: \"kubernetes.io/projected/90102371-81a6-4419-a23e-72f0ba01c595-kube-api-access-t6869\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.973765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973747    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.973841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973810    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.973897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973853    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90102371-81a6-4419-a23e-72f0ba01c595-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.973967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.973905    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:08.974044 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:08.974018    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.074799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.074760    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90102371-81a6-4419-a23e-72f0ba01c595-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.074799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.074801    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6869\" (UniqueName: \"kubernetes.io/projected/90102371-81a6-4419-a23e-72f0ba01c595-kube-api-access-t6869\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.074818    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.074833    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.074857    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90102371-81a6-4419-a23e-72f0ba01c595-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.074918    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.074956    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.075022    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.075052    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90102371-81a6-4419-a23e-72f0ba01c595-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.075287    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-istio-data\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.075379    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-workload-certs\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.075428    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-credential-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075603 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.075550    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90102371-81a6-4419-a23e-72f0ba01c595-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.075603 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.075566    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-workload-socket\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.077286 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.077266    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90102371-81a6-4419-a23e-72f0ba01c595-istio-envoy\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.077403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.077387    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90102371-81a6-4419-a23e-72f0ba01c595-istio-podinfo\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.100969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.100941    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90102371-81a6-4419-a23e-72f0ba01c595-istio-token\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.101872 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.101846    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6869\" (UniqueName: \"kubernetes.io/projected/90102371-81a6-4419-a23e-72f0ba01c595-kube-api-access-t6869\") pod \"maas-default-gateway-openshift-default-746ff8f49f-7bzbg\" (UID: \"90102371-81a6-4419-a23e-72f0ba01c595\") " pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.206060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.206024    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:09.531477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.531445    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"]
Oct 29 23:36:09.531627 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:36:09.531485    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90102371_81a6_4419_a23e_72f0ba01c595.slice/crio-2b7208218c71c79d83d71cd504d9e62c913d93545247e71767d64403defaec71 WatchSource:0}: Error finding container 2b7208218c71c79d83d71cd504d9e62c913d93545247e71767d64403defaec71: Status 404 returned error can't find the container with id 2b7208218c71c79d83d71cd504d9e62c913d93545247e71767d64403defaec71
Oct 29 23:36:09.533631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.533600    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:36:09.533701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.533663    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:36:09.533701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:09.533692    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:36:09.924749 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:36:09.924713    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:36:10.312951 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:10.312845    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" event={"ID":"90102371-81a6-4419-a23e-72f0ba01c595","Type":"ContainerStarted","Data":"60456177caa302139fbb142873deee34c78aafc3470b4a5cc8d293dee2f59711"}
Oct 29 23:36:10.312951 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:10.312924    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" event={"ID":"90102371-81a6-4419-a23e-72f0ba01c595","Type":"ContainerStarted","Data":"2b7208218c71c79d83d71cd504d9e62c913d93545247e71767d64403defaec71"}
Oct 29 23:36:10.389978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:10.389925    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podStartSLOduration=2.3899099440000002 podStartE2EDuration="2.389909944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:10.38727762 +0000 UTC m=+1348.491538006" watchObservedRunningTime="2026-10-29 23:36:10.389909944 +0000 UTC m=+1348.494170404"
Oct 29 23:36:11.206786 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:11.206749    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:11.208228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:11.208204    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 23:36:11.208303 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:11.208254    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 23:36:12.207014 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:12.206977    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 23:36:12.207407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:12.207032    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 23:36:13.206642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:13.206605    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 23:36:13.206814 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:13.206658    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 23:36:14.207279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.207242    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 23:36:14.207783 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.207304    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 23:36:14.889982 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.889943    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"]
Oct 29 23:36:14.893902 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.893873    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:14.912236 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.912205    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:36:14.912441 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.912312    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:36:14.913689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.913665    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:36:14.916337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.916308    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"]
Oct 29 23:36:14.925857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:14.925824    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:36:15.054594 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.054547    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.054594 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.054599    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.054826 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.054696    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr4td\" (UniqueName: \"kubernetes.io/projected/daca662d-991e-4dcf-8c31-fc221e324171-kube-api-access-dr4td\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.156003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.155858    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.156003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.155964    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.156234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.156028    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr4td\" (UniqueName: \"kubernetes.io/projected/daca662d-991e-4dcf-8c31-fc221e324171-kube-api-access-dr4td\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.156663 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.156634    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.158334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.158315    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.182310 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.182280    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr4td\" (UniqueName: \"kubernetes.io/projected/daca662d-991e-4dcf-8c31-fc221e324171-kube-api-access-dr4td\") pod \"maas-api-revoke-sub-85237e45-cl2n5\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.207323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.207294    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 23:36:15.207694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.207343    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 23:36:15.207694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.207658    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:15.537662 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:15.537630    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"]
Oct 29 23:36:15.538351 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:36:15.538323    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaca662d_991e_4dcf_8c31_fc221e324171.slice/crio-3e6dede39b70c9a85162a08580c1029c2ec5dfd870c8ba31d04ccf5157539890 WatchSource:0}: Error finding container 3e6dede39b70c9a85162a08580c1029c2ec5dfd870c8ba31d04ccf5157539890: Status 404 returned error can't find the container with id 3e6dede39b70c9a85162a08580c1029c2ec5dfd870c8ba31d04ccf5157539890
Oct 29 23:36:16.207131 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:16.207096    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 29 23:36:16.207300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:16.207150    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 29 23:36:16.379752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:16.379707    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5" event={"ID":"daca662d-991e-4dcf-8c31-fc221e324171","Type":"ContainerStarted","Data":"f4de73d10c12834de0ae9153ed621e62e5a0bb5a5e6f6c8d54393b8bce614214"}
Oct 29 23:36:16.380168 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:16.379770    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5" event={"ID":"daca662d-991e-4dcf-8c31-fc221e324171","Type":"ContainerStarted","Data":"3e6dede39b70c9a85162a08580c1029c2ec5dfd870c8ba31d04ccf5157539890"}
Oct 29 23:36:16.428907 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:16.428851    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5" podStartSLOduration=2.428835747 podStartE2EDuration="2.428835747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:36:16.427161834 +0000 UTC m=+1354.531422220" watchObservedRunningTime="2026-10-29 23:36:16.428835747 +0000 UTC m=+1354.533096132"
Oct 29 23:36:17.390183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:17.390136    6203 generic.go:356] "Generic (PLEG): container finished" podID="daca662d-991e-4dcf-8c31-fc221e324171" containerID="f4de73d10c12834de0ae9153ed621e62e5a0bb5a5e6f6c8d54393b8bce614214" exitCode=0
Oct 29 23:36:17.390597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:17.390200    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5" event={"ID":"daca662d-991e-4dcf-8c31-fc221e324171","Type":"ContainerDied","Data":"f4de73d10c12834de0ae9153ed621e62e5a0bb5a5e6f6c8d54393b8bce614214"}
Oct 29 23:36:18.210961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.210919    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:36:18.211156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.210980    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:36:18.557443 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.557410    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:18.597245 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.597214    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daca662d-991e-4dcf-8c31-fc221e324171-kube-api-access-dr4td\" (UniqueName: \"kubernetes.io/projected/daca662d-991e-4dcf-8c31-fc221e324171-kube-api-access-dr4td\") pod \"daca662d-991e-4dcf-8c31-fc221e324171\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") "
Oct 29 23:36:18.597426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.597281    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-xks-ca\") pod \"daca662d-991e-4dcf-8c31-fc221e324171\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") "
Oct 29 23:36:18.597426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.597351    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-openshift-ca\") pod \"daca662d-991e-4dcf-8c31-fc221e324171\" (UID: \"daca662d-991e-4dcf-8c31-fc221e324171\") "
Oct 29 23:36:18.597735 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.597705    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-openshift-ca" pod "daca662d-991e-4dcf-8c31-fc221e324171" (UID: "daca662d-991e-4dcf-8c31-fc221e324171"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:36:18.599429 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.599403    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-xks-ca" pod "daca662d-991e-4dcf-8c31-fc221e324171" (UID: "daca662d-991e-4dcf-8c31-fc221e324171"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:36:18.599542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.599446    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daca662d-991e-4dcf-8c31-fc221e324171-kube-api-access-dr4td" pod "daca662d-991e-4dcf-8c31-fc221e324171" (UID: "daca662d-991e-4dcf-8c31-fc221e324171"). InnerVolumeSpecName "kube-api-access-dr4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:36:18.698718 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.698679    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr4td\" (UniqueName: \"kubernetes.io/projected/daca662d-991e-4dcf-8c31-fc221e324171-kube-api-access-dr4td\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:36:18.698718 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.698710    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:36:18.698718 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:18.698723    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daca662d-991e-4dcf-8c31-fc221e324171-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:36:19.252119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:19.252065    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:36:19.252351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:19.252135    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:36:19.407114 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:19.407081    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"
Oct 29 23:36:19.407277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:19.407076    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5" event={"ID":"daca662d-991e-4dcf-8c31-fc221e324171","Type":"ContainerDied","Data":"3e6dede39b70c9a85162a08580c1029c2ec5dfd870c8ba31d04ccf5157539890"}
Oct 29 23:36:19.407277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:19.407196    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6dede39b70c9a85162a08580c1029c2ec5dfd870c8ba31d04ccf5157539890"
Oct 29 23:36:20.033182 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:20.033140    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"]
Oct 29 23:36:20.054900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:20.054212    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d05e74c-62v69"]
Oct 29 23:36:20.105349 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:20.105309    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:20.105841 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:20.105815    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:20.106648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:20.106622    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg"
Oct 29 23:36:22.018321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:22.018290    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5121fa-65a7-4ca8-afd9-7615d5e0276f" path="/var/lib/kubelet/pods/0c5121fa-65a7-4ca8-afd9-7615d5e0276f/volumes"
Oct 29 23:36:31.032761 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:31.032671    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"]
Oct 29 23:36:31.047050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:31.047020    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e28a881-xbwnn"]
Oct 29 23:36:32.017557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:32.017513    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155fe462-1192-4c66-ac31-d4ec53e5b29e" path="/var/lib/kubelet/pods/155fe462-1192-4c66-ac31-d4ec53e5b29e/volumes"
Oct 29 23:36:37.031733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:37.031697    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"]
Oct 29 23:36:37.046180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:37.046153    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3c9ebb-949tb"]
Oct 29 23:36:38.016725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:38.016688    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e11d63-f475-428e-a5d9-6ad78d4b35ac" path="/var/lib/kubelet/pods/e1e11d63-f475-428e-a5d9-6ad78d4b35ac/volumes"
Oct 29 23:36:39.032644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:39.032607    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"]
Oct 29 23:36:39.047133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:39.047101    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a20e476d-7fxcd"]
Oct 29 23:36:39.923494 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:36:39.923457    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:36:40.017010 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:40.016980    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f622b3-5c02-40f0-aeca-7dcf14a8b0f6" path="/var/lib/kubelet/pods/31f622b3-5c02-40f0-aeca-7dcf14a8b0f6/volumes"
Oct 29 23:36:45.030962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:45.030927    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"]
Oct 29 23:36:45.045058 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:45.045027    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276d646a-tlm6s"]
Oct 29 23:36:46.017764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:46.017725    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe9bc56-3e57-4bb7-8144-5dbca79e7379" path="/var/lib/kubelet/pods/efe9bc56-3e57-4bb7-8144-5dbca79e7379/volumes"
Oct 29 23:36:46.790521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:46.790490    6203 scope.go:122] "RemoveContainer" containerID="4344196f3455dbe86f33e449495a004a7eec5c1e134b1a95a6804ecf8576afde"
Oct 29 23:36:46.800308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:46.800281    6203 scope.go:122] "RemoveContainer" containerID="a7786a390e37b385949ef1cc5e1cefedf902e25db33173a10be958d2b9b67d7b"
Oct 29 23:36:46.809337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:46.809318    6203 scope.go:122] "RemoveContainer" containerID="a687a9996c1376b3a03e3b7af5de973615c49f37c9dc496e59d7df17d48554e3"
Oct 29 23:36:46.817753 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:46.817732    6203 scope.go:122] "RemoveContainer" containerID="ad72fa685023ca43caa54b35b5dbc3a36c09921733e62721e8f4c58ec3606446"
Oct 29 23:36:46.827183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:46.827157    6203 scope.go:122] "RemoveContainer" containerID="6c81a697b56b31c8c1df032108147f7330349db481c40a2c17392cd6c53d78c8"
Oct 29 23:36:54.038403 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:54.038371    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"]
Oct 29 23:36:54.052649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:54.052619    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e14aef47-sm2rd"]
Oct 29 23:36:55.032174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:55.032136    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"]
Oct 29 23:36:55.034468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:55.034441    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dbae4be-r5ntw"]
Oct 29 23:36:56.017229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:56.017197    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d1ab42-5bf8-48a5-8988-e2f8729911ab" path="/var/lib/kubelet/pods/72d1ab42-5bf8-48a5-8988-e2f8729911ab/volumes"
Oct 29 23:36:56.017623 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:56.017599    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c499ac31-e352-431e-88e0-32df1bd0d90e" path="/var/lib/kubelet/pods/c499ac31-e352-431e-88e0-32df1bd0d90e/volumes"
Oct 29 23:36:58.034598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:58.034566    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"]
Oct 29 23:36:58.048615 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:58.048586    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb669009-d4rh7"]
Oct 29 23:36:59.030672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:59.030635    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"]
Oct 29 23:36:59.044941 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:36:59.044911    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5026caab-8ngvg"]
Oct 29 23:37:00.019434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:00.019396    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b580b2e2-83b9-44b0-b807-c71126fd8864" path="/var/lib/kubelet/pods/b580b2e2-83b9-44b0-b807-c71126fd8864/volumes"
Oct 29 23:37:00.019808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:00.019792    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1fa1fa-c139-4b83-bb80-f18e1c104860" path="/var/lib/kubelet/pods/ba1fa1fa-c139-4b83-bb80-f18e1c104860/volumes"
Oct 29 23:37:02.023042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:02.023001    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:37:02.023483 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:02.023050    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:37:05.031459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.031420    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"]
Oct 29 23:37:05.045885 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.045850    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"]
Oct 29 23:37:05.047949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.047918    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fb483b5-5w49m"]
Oct 29 23:37:05.049770 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.049751    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d73b451a-qbrnf"]
Oct 29 23:37:05.141118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.141086    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"]
Oct 29 23:37:05.141600 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.141585    6203 cpu_manager.go:414] "containerMap: removing container" podUID="daca662d-991e-4dcf-8c31-fc221e324171" containerName="revoke-keys"
Oct 29 23:37:05.141643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.141602    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daca662d-991e-4dcf-8c31-fc221e324171" containerName="revoke-keys"
Oct 29 23:37:05.141681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.141676    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="daca662d-991e-4dcf-8c31-fc221e324171" containerName="revoke-keys"
Oct 29 23:37:05.144815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.144799    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.162946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.162920    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:37:05.163127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.162936    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:37:05.163127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.162968    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:37:05.163256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.163011    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:37:05.166385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.166360    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"]
Oct 29 23:37:05.249957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.249865    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxl7t\" (UniqueName: \"kubernetes.io/projected/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-kube-api-access-bxl7t\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.250160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.250038    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.250160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.250081    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.351573 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.351487    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxl7t\" (UniqueName: \"kubernetes.io/projected/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-kube-api-access-bxl7t\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.351720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.351583    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.351720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.351628    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.352255 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.352229    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.353973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.353944    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.377961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.377931    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxl7t\" (UniqueName: \"kubernetes.io/projected/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-kube-api-access-bxl7t\") pod \"maas-api-revoke-sub-c1e271a9-ghn86\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.457886 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.457837    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:05.757396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:05.757365    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"]
Oct 29 23:37:05.757798 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:37:05.757775    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed3c848_038c_453a_8b66_6fb68d2d8dc4.slice/crio-8bba1416164a975471f4466d6d1606d1190f3ac83cc167ee83e01d122e8f10a5 WatchSource:0}: Error finding container 8bba1416164a975471f4466d6d1606d1190f3ac83cc167ee83e01d122e8f10a5: Status 404 returned error can't find the container with id 8bba1416164a975471f4466d6d1606d1190f3ac83cc167ee83e01d122e8f10a5
Oct 29 23:37:06.017094 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:06.017013    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a19dd1e-502f-4d23-8666-a05938cef188" path="/var/lib/kubelet/pods/8a19dd1e-502f-4d23-8666-a05938cef188/volumes"
Oct 29 23:37:06.017458 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:06.017443    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39c2cc6-0db7-4b32-9786-f5b911fa6e52" path="/var/lib/kubelet/pods/b39c2cc6-0db7-4b32-9786-f5b911fa6e52/volumes"
Oct 29 23:37:06.741522 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:06.741480    6203 generic.go:356] "Generic (PLEG): container finished" podID="1ed3c848-038c-453a-8b66-6fb68d2d8dc4" containerID="8e4364942cc9a21d8d3307b3d61e433763680dd878644e11aac825b49303303a" exitCode=0
Oct 29 23:37:06.741950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:06.741558    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86" event={"ID":"1ed3c848-038c-453a-8b66-6fb68d2d8dc4","Type":"ContainerDied","Data":"8e4364942cc9a21d8d3307b3d61e433763680dd878644e11aac825b49303303a"}
Oct 29 23:37:06.741950 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:06.741606    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86" event={"ID":"1ed3c848-038c-453a-8b66-6fb68d2d8dc4","Type":"ContainerStarted","Data":"8bba1416164a975471f4466d6d1606d1190f3ac83cc167ee83e01d122e8f10a5"}
Oct 29 23:37:07.872661 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:07.872634    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:07.978195 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:07.978158    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-xks-ca\") pod \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") "
Oct 29 23:37:07.978398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:07.978233    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-openshift-ca\") pod \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") "
Oct 29 23:37:07.978398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:07.978308    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-kube-api-access-bxl7t\" (UniqueName: \"kubernetes.io/projected/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-kube-api-access-bxl7t\") pod \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\" (UID: \"1ed3c848-038c-453a-8b66-6fb68d2d8dc4\") "
Oct 29 23:37:07.978605 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:07.978579    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-openshift-ca" pod "1ed3c848-038c-453a-8b66-6fb68d2d8dc4" (UID: "1ed3c848-038c-453a-8b66-6fb68d2d8dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:37:07.980333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:07.980311    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-xks-ca" pod "1ed3c848-038c-453a-8b66-6fb68d2d8dc4" (UID: "1ed3c848-038c-453a-8b66-6fb68d2d8dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:37:07.980451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:07.980433    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-kube-api-access-bxl7t" pod "1ed3c848-038c-453a-8b66-6fb68d2d8dc4" (UID: "1ed3c848-038c-453a-8b66-6fb68d2d8dc4"). InnerVolumeSpecName "kube-api-access-bxl7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:37:08.079186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:08.079151    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:08.079186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:08.079181    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:08.079186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:08.079191    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxl7t\" (UniqueName: \"kubernetes.io/projected/1ed3c848-038c-453a-8b66-6fb68d2d8dc4-kube-api-access-bxl7t\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:08.761528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:08.761496    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"
Oct 29 23:37:08.761528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:08.761512    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86" event={"ID":"1ed3c848-038c-453a-8b66-6fb68d2d8dc4","Type":"ContainerDied","Data":"8bba1416164a975471f4466d6d1606d1190f3ac83cc167ee83e01d122e8f10a5"}
Oct 29 23:37:08.761740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:08.761577    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bba1416164a975471f4466d6d1606d1190f3ac83cc167ee83e01d122e8f10a5"
Oct 29 23:37:09.032989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:09.032869    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"]
Oct 29 23:37:09.035382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:09.035350    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc838e28-nnfnj"]
Oct 29 23:37:09.924422 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:37:09.924386    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:37:10.018391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:10.018354    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff80c93-47a4-4563-9af9-93cf16356b51" path="/var/lib/kubelet/pods/dff80c93-47a4-4563-9af9-93cf16356b51/volumes"
Oct 29 23:37:20.033349 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:20.033313    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"]
Oct 29 23:37:20.047652 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:20.047622    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68ddf24a-hl9bs"]
Oct 29 23:37:22.020921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:22.020855    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff3ed99-48d3-4d2d-84fa-e23ffa720a08" path="/var/lib/kubelet/pods/9ff3ed99-48d3-4d2d-84fa-e23ffa720a08/volumes"
Oct 29 23:37:22.053246 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:22.053206    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"]
Oct 29 23:37:22.073603 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:22.073565    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a06a50-t7q6s"]
Oct 29 23:37:24.017570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:24.017531    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf91e33-927c-4888-a6b8-1c5875e673a6" path="/var/lib/kubelet/pods/7bf91e33-927c-4888-a6b8-1c5875e673a6/volumes"
Oct 29 23:37:25.028142 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:25.028102    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"]
Oct 29 23:37:25.042670 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:25.042644    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106efa1b-jbrcx"]
Oct 29 23:37:26.016889 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:26.016848    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61de0dcd-e20f-4365-91a8-e4bd91b674a2" path="/var/lib/kubelet/pods/61de0dcd-e20f-4365-91a8-e4bd91b674a2/volumes"
Oct 29 23:37:28.137761 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.137724    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"]
Oct 29 23:37:28.138200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.138170    6203 cpu_manager.go:414] "containerMap: removing container" podUID="1ed3c848-038c-453a-8b66-6fb68d2d8dc4" containerName="revoke-keys"
Oct 29 23:37:28.138200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.138182    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed3c848-038c-453a-8b66-6fb68d2d8dc4" containerName="revoke-keys"
Oct 29 23:37:28.138284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.138266    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed3c848-038c-453a-8b66-6fb68d2d8dc4" containerName="revoke-keys"
Oct 29 23:37:28.144747 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.144704    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.161639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.161609    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"]
Oct 29 23:37:28.173708 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.173684    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:37:28.173708 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.173684    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:37:28.173867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.173733    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:37:28.176723 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.176703    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:37:28.220981 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.220946    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26m4\" (UniqueName: \"kubernetes.io/projected/8e192588-39c1-461b-92e7-26a8f05a4f17-kube-api-access-j26m4\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.220981 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.220984    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.221211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.221070    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.322506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.322474    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26m4\" (UniqueName: \"kubernetes.io/projected/8e192588-39c1-461b-92e7-26a8f05a4f17-kube-api-access-j26m4\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.322506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.322505    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.322731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.322528    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.323154 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.323136    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.324830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.324797    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.349287 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.349257    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26m4\" (UniqueName: \"kubernetes.io/projected/8e192588-39c1-461b-92e7-26a8f05a4f17-kube-api-access-j26m4\") pod \"maas-api-revoke-sub-2e1f5b5f-k7vqh\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.459654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.459622    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:28.759995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.759961    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"]
Oct 29 23:37:28.760496 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:37:28.760469    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e192588_39c1_461b_92e7_26a8f05a4f17.slice/crio-61235e7aa72b03f81ebae2e36658b04b6418b4e9a80777f25ca8a69318fc7760 WatchSource:0}: Error finding container 61235e7aa72b03f81ebae2e36658b04b6418b4e9a80777f25ca8a69318fc7760: Status 404 returned error can't find the container with id 61235e7aa72b03f81ebae2e36658b04b6418b4e9a80777f25ca8a69318fc7760
Oct 29 23:37:28.945636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:28.945603    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh" event={"ID":"8e192588-39c1-461b-92e7-26a8f05a4f17","Type":"ContainerStarted","Data":"61235e7aa72b03f81ebae2e36658b04b6418b4e9a80777f25ca8a69318fc7760"}
Oct 29 23:37:29.953111 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:29.953077    6203 generic.go:356] "Generic (PLEG): container finished" podID="8e192588-39c1-461b-92e7-26a8f05a4f17" containerID="204e737c00f71d856d973387690f0665255d4a452067132dd4d5d4cef7c5903f" exitCode=0
Oct 29 23:37:29.953511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:29.953169    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh" event={"ID":"8e192588-39c1-461b-92e7-26a8f05a4f17","Type":"ContainerDied","Data":"204e737c00f71d856d973387690f0665255d4a452067132dd4d5d4cef7c5903f"}
Oct 29 23:37:31.092470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.092443    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:31.251449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.251362    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e192588-39c1-461b-92e7-26a8f05a4f17-kube-api-access-j26m4\" (UniqueName: \"kubernetes.io/projected/8e192588-39c1-461b-92e7-26a8f05a4f17-kube-api-access-j26m4\") pod \"8e192588-39c1-461b-92e7-26a8f05a4f17\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") "
Oct 29 23:37:31.251449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.251433    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-xks-ca\") pod \"8e192588-39c1-461b-92e7-26a8f05a4f17\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") "
Oct 29 23:37:31.251695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.251472    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-openshift-ca\") pod \"8e192588-39c1-461b-92e7-26a8f05a4f17\" (UID: \"8e192588-39c1-461b-92e7-26a8f05a4f17\") "
Oct 29 23:37:31.251870 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.251839    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-openshift-ca" pod "8e192588-39c1-461b-92e7-26a8f05a4f17" (UID: "8e192588-39c1-461b-92e7-26a8f05a4f17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:37:31.251996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.251943    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:31.253563 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.253539    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-xks-ca" pod "8e192588-39c1-461b-92e7-26a8f05a4f17" (UID: "8e192588-39c1-461b-92e7-26a8f05a4f17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:37:31.253640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.253537    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e192588-39c1-461b-92e7-26a8f05a4f17-kube-api-access-j26m4" pod "8e192588-39c1-461b-92e7-26a8f05a4f17" (UID: "8e192588-39c1-461b-92e7-26a8f05a4f17"). InnerVolumeSpecName "kube-api-access-j26m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:37:31.353016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.352982    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e192588-39c1-461b-92e7-26a8f05a4f17-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:31.353016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.353010    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26m4\" (UniqueName: \"kubernetes.io/projected/8e192588-39c1-461b-92e7-26a8f05a4f17-kube-api-access-j26m4\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:31.975745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.975709    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"
Oct 29 23:37:31.975745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.975730    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh" event={"ID":"8e192588-39c1-461b-92e7-26a8f05a4f17","Type":"ContainerDied","Data":"61235e7aa72b03f81ebae2e36658b04b6418b4e9a80777f25ca8a69318fc7760"}
Oct 29 23:37:31.976003 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:31.975789    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61235e7aa72b03f81ebae2e36658b04b6418b4e9a80777f25ca8a69318fc7760"
Oct 29 23:37:33.030961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:33.030930    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"]
Oct 29 23:37:33.045225 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:33.045193    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00c79458-j2ssb"]
Oct 29 23:37:34.017525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:34.017483    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fb5de7-9a1e-4f50-8abd-33a64a830065" path="/var/lib/kubelet/pods/19fb5de7-9a1e-4f50-8abd-33a64a830065/volumes"
Oct 29 23:37:39.423427 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.423389    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"]
Oct 29 23:37:39.423934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.423828    6203 cpu_manager.go:414] "containerMap: removing container" podUID="8e192588-39c1-461b-92e7-26a8f05a4f17" containerName="revoke-keys"
Oct 29 23:37:39.423934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.423840    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e192588-39c1-461b-92e7-26a8f05a4f17" containerName="revoke-keys"
Oct 29 23:37:39.424016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.423956    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e192588-39c1-461b-92e7-26a8f05a4f17" containerName="revoke-keys"
Oct 29 23:37:39.427376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.427359    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.436358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.436329    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.436557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.436537    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.436597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.436583    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7xd\" (UniqueName: \"kubernetes.io/projected/b0970c30-301a-420b-b7e4-87b7ad4200fc-kube-api-access-fw7xd\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.446989 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.446965    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"]
Oct 29 23:37:39.458972 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.458945    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:37:39.458972 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.458953    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:37:39.459144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.459013    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:37:39.459144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.459031    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:37:39.537780 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.537747    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.538012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.537849    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.538012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.537886    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7xd\" (UniqueName: \"kubernetes.io/projected/b0970c30-301a-420b-b7e4-87b7ad4200fc-kube-api-access-fw7xd\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.538623 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.538595    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.540197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.540175    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.563688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.563652    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7xd\" (UniqueName: \"kubernetes.io/projected/b0970c30-301a-420b-b7e4-87b7ad4200fc-kube-api-access-fw7xd\") pod \"maas-api-revoke-sub-ec843018-pcs7d\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.740803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:39.740716    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:39.922439 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:37:39.922400    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:37:40.056256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:40.056216    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"]
Oct 29 23:37:40.056978 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:37:40.056956    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0970c30_301a_420b_b7e4_87b7ad4200fc.slice/crio-a0febf61b8a6a0df73eafc4f0a288a0d12aeec905019602d80d5f480334ea8ee WatchSource:0}: Error finding container a0febf61b8a6a0df73eafc4f0a288a0d12aeec905019602d80d5f480334ea8ee: Status 404 returned error can't find the container with id a0febf61b8a6a0df73eafc4f0a288a0d12aeec905019602d80d5f480334ea8ee
Oct 29 23:37:41.029542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:41.029505    6203 generic.go:356] "Generic (PLEG): container finished" podID="b0970c30-301a-420b-b7e4-87b7ad4200fc" containerID="98bc93739a72ede8714404d8b085dfa19e5299cd17372fa11d1b971750906a5a" exitCode=0
Oct 29 23:37:41.030001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:41.029594    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d" event={"ID":"b0970c30-301a-420b-b7e4-87b7ad4200fc","Type":"ContainerDied","Data":"98bc93739a72ede8714404d8b085dfa19e5299cd17372fa11d1b971750906a5a"}
Oct 29 23:37:41.030001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:41.029658    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d" event={"ID":"b0970c30-301a-420b-b7e4-87b7ad4200fc","Type":"ContainerStarted","Data":"a0febf61b8a6a0df73eafc4f0a288a0d12aeec905019602d80d5f480334ea8ee"}
Oct 29 23:37:42.195867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.195817    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:42.263461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.263407    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0970c30-301a-420b-b7e4-87b7ad4200fc-kube-api-access-fw7xd\" (UniqueName: \"kubernetes.io/projected/b0970c30-301a-420b-b7e4-87b7ad4200fc-kube-api-access-fw7xd\") pod \"b0970c30-301a-420b-b7e4-87b7ad4200fc\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") "
Oct 29 23:37:42.263672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.263487    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-openshift-ca\") pod \"b0970c30-301a-420b-b7e4-87b7ad4200fc\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") "
Oct 29 23:37:42.263672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.263535    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-xks-ca\") pod \"b0970c30-301a-420b-b7e4-87b7ad4200fc\" (UID: \"b0970c30-301a-420b-b7e4-87b7ad4200fc\") "
Oct 29 23:37:42.264324 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.264291    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-openshift-ca" pod "b0970c30-301a-420b-b7e4-87b7ad4200fc" (UID: "b0970c30-301a-420b-b7e4-87b7ad4200fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:37:42.266642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.266611    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-xks-ca" pod "b0970c30-301a-420b-b7e4-87b7ad4200fc" (UID: "b0970c30-301a-420b-b7e4-87b7ad4200fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:37:42.266764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.266729    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0970c30-301a-420b-b7e4-87b7ad4200fc-kube-api-access-fw7xd" pod "b0970c30-301a-420b-b7e4-87b7ad4200fc" (UID: "b0970c30-301a-420b-b7e4-87b7ad4200fc"). InnerVolumeSpecName "kube-api-access-fw7xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:37:42.364770 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.364676    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7xd\" (UniqueName: \"kubernetes.io/projected/b0970c30-301a-420b-b7e4-87b7ad4200fc-kube-api-access-fw7xd\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:42.364770 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.364719    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:42.364770 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:42.364734    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0970c30-301a-420b-b7e4-87b7ad4200fc-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:37:43.047653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:43.047624    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"
Oct 29 23:37:43.047988 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:43.047651    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d" event={"ID":"b0970c30-301a-420b-b7e4-87b7ad4200fc","Type":"ContainerDied","Data":"a0febf61b8a6a0df73eafc4f0a288a0d12aeec905019602d80d5f480334ea8ee"}
Oct 29 23:37:43.048139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:43.048018    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0febf61b8a6a0df73eafc4f0a288a0d12aeec905019602d80d5f480334ea8ee"
Oct 29 23:37:46.901815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:46.901785    6203 scope.go:122] "RemoveContainer" containerID="2f8923434589808294adc9b2d29b0c0316a4519ba32b91839d683e64cffbcb44"
Oct 29 23:37:46.912070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:46.912044    6203 scope.go:122] "RemoveContainer" containerID="2912293498a1f72df0cad8dc094e2ed7c80564dfdc68c0417ce7d50016d42d63"
Oct 29 23:37:46.920848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:46.920823    6203 scope.go:122] "RemoveContainer" containerID="e9c06ef9ca789f8bc2e54cdd6cf1a4c6b032cd6a705a90e81563be867d25487a"
Oct 29 23:37:46.929192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:46.929167    6203 scope.go:122] "RemoveContainer" containerID="e2bf68a8a87670eac622ebce8366f86ca108cc0ebf9703486eff1df4d8b037d3"
Oct 29 23:37:46.937283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:46.937264    6203 scope.go:122] "RemoveContainer" containerID="bad5ddcbfdc45b5b001ee407e22e7fe7257698db5faacb1f611ab8931d30b026"
Oct 29 23:37:49.030304 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:49.030270    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"]
Oct 29 23:37:49.045694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:49.045657    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47fc6e21-xg97k"]
Oct 29 23:37:50.017010 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:37:50.016968    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c922007-198e-4db9-bb6a-013860a92683" path="/var/lib/kubelet/pods/8c922007-198e-4db9-bb6a-013860a92683/volumes"
Oct 29 23:38:01.032134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:01.032090    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"]
Oct 29 23:38:01.046642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:01.046609    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"]
Oct 29 23:38:01.048687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:01.048667    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e8d635-58hpd"]
Oct 29 23:38:01.050348 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:01.050327    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf3321-8xhbr"]
Oct 29 23:38:02.016865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:02.016833    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0616a5-b652-4341-9c6e-45c90c51b10f" path="/var/lib/kubelet/pods/6a0616a5-b652-4341-9c6e-45c90c51b10f/volumes"
Oct 29 23:38:02.017250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:02.017236    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f9bf78-eed7-4beb-a4eb-61d51a104f41" path="/var/lib/kubelet/pods/e4f9bf78-eed7-4beb-a4eb-61d51a104f41/volumes"
Oct 29 23:38:05.030115 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:05.030077    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"]
Oct 29 23:38:05.044620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:05.044590    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65701252-ffwwh"]
Oct 29 23:38:06.022273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:06.022229    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e" path="/var/lib/kubelet/pods/b83f02cd-7523-4bbf-8ee9-56cd9e5dbf5e/volumes"
Oct 29 23:38:07.151460 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.151417    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"]
Oct 29 23:38:07.152061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.152040    6203 cpu_manager.go:414] "containerMap: removing container" podUID="b0970c30-301a-420b-b7e4-87b7ad4200fc" containerName="revoke-keys"
Oct 29 23:38:07.152170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.152063    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0970c30-301a-420b-b7e4-87b7ad4200fc" containerName="revoke-keys"
Oct 29 23:38:07.152170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.152164    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0970c30-301a-420b-b7e4-87b7ad4200fc" containerName="revoke-keys"
Oct 29 23:38:07.155539 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.155516    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.173702 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.173676    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:38:07.188034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.188006    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:38:07.188190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.188068    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:38:07.188190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.188145    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:38:07.190581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.190562    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"]
Oct 29 23:38:07.198977 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.198946    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.199150 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.199047    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md8jh\" (UniqueName: \"kubernetes.io/projected/2207d0a2-2213-45a0-9fdf-c4666729bea7-kube-api-access-md8jh\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.199252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.199154    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.299664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.299628    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.299857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.299690    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.299857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.299742    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md8jh\" (UniqueName: \"kubernetes.io/projected/2207d0a2-2213-45a0-9fdf-c4666729bea7-kube-api-access-md8jh\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.300369 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.300340    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.302119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.302092    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.325556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.325524    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md8jh\" (UniqueName: \"kubernetes.io/projected/2207d0a2-2213-45a0-9fdf-c4666729bea7-kube-api-access-md8jh\") pod \"maas-api-revoke-sub-046e1343-4wj4m\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.468476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.468446    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:07.768837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:07.768807    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"]
Oct 29 23:38:07.770304 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:38:07.770260    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2207d0a2_2213_45a0_9fdf_c4666729bea7.slice/crio-4e1ad9fef77c722cd209f73ba0819819450a0ad6cf4aae942b91e6aae79e7776 WatchSource:0}: Error finding container 4e1ad9fef77c722cd209f73ba0819819450a0ad6cf4aae942b91e6aae79e7776: Status 404 returned error can't find the container with id 4e1ad9fef77c722cd209f73ba0819819450a0ad6cf4aae942b91e6aae79e7776
Oct 29 23:38:08.030648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:08.030613    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"]
Oct 29 23:38:08.046129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:08.046104    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b27806e-kjznm"]
Oct 29 23:38:08.271482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:08.271445    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m" event={"ID":"2207d0a2-2213-45a0-9fdf-c4666729bea7","Type":"ContainerStarted","Data":"c0aa50e3e907647b68bf2565de044380983b2ba26ce4d3f57ade01c9478312ee"}
Oct 29 23:38:08.271866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:08.271504    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m" event={"ID":"2207d0a2-2213-45a0-9fdf-c4666729bea7","Type":"ContainerStarted","Data":"4e1ad9fef77c722cd209f73ba0819819450a0ad6cf4aae942b91e6aae79e7776"}
Oct 29 23:38:08.316931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:08.316819    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m" podStartSLOduration=1.316806687 podStartE2EDuration="1.316806687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:08.316145494 +0000 UTC m=+1466.420405863" watchObservedRunningTime="2026-10-29 23:38:08.316806687 +0000 UTC m=+1466.421067075"
Oct 29 23:38:09.283372 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:09.283333    6203 generic.go:356] "Generic (PLEG): container finished" podID="2207d0a2-2213-45a0-9fdf-c4666729bea7" containerID="c0aa50e3e907647b68bf2565de044380983b2ba26ce4d3f57ade01c9478312ee" exitCode=0
Oct 29 23:38:09.283801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:09.283422    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m" event={"ID":"2207d0a2-2213-45a0-9fdf-c4666729bea7","Type":"ContainerDied","Data":"c0aa50e3e907647b68bf2565de044380983b2ba26ce4d3f57ade01c9478312ee"}
Oct 29 23:38:09.923487 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:38:09.923451    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:38:10.017967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.017937    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2886a9-9a5a-4044-aa00-2b65110ccd6f" path="/var/lib/kubelet/pods/8c2886a9-9a5a-4044-aa00-2b65110ccd6f/volumes"
Oct 29 23:38:10.430811 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.430781    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:10.524545 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.524510    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-openshift-ca\") pod \"2207d0a2-2213-45a0-9fdf-c4666729bea7\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") "
Oct 29 23:38:10.524830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.524569    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2207d0a2-2213-45a0-9fdf-c4666729bea7-kube-api-access-md8jh\" (UniqueName: \"kubernetes.io/projected/2207d0a2-2213-45a0-9fdf-c4666729bea7-kube-api-access-md8jh\") pod \"2207d0a2-2213-45a0-9fdf-c4666729bea7\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") "
Oct 29 23:38:10.524830 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.524673    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-xks-ca\") pod \"2207d0a2-2213-45a0-9fdf-c4666729bea7\" (UID: \"2207d0a2-2213-45a0-9fdf-c4666729bea7\") "
Oct 29 23:38:10.524994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.524947    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-openshift-ca" pod "2207d0a2-2213-45a0-9fdf-c4666729bea7" (UID: "2207d0a2-2213-45a0-9fdf-c4666729bea7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:38:10.526616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.526596    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-xks-ca" pod "2207d0a2-2213-45a0-9fdf-c4666729bea7" (UID: "2207d0a2-2213-45a0-9fdf-c4666729bea7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:38:10.526693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.526673    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2207d0a2-2213-45a0-9fdf-c4666729bea7-kube-api-access-md8jh" pod "2207d0a2-2213-45a0-9fdf-c4666729bea7" (UID: "2207d0a2-2213-45a0-9fdf-c4666729bea7"). InnerVolumeSpecName "kube-api-access-md8jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:38:10.625777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.625703    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md8jh\" (UniqueName: \"kubernetes.io/projected/2207d0a2-2213-45a0-9fdf-c4666729bea7-kube-api-access-md8jh\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:10.625777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.625732    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:10.625777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:10.625743    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2207d0a2-2213-45a0-9fdf-c4666729bea7-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:11.304038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:11.304002    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m" event={"ID":"2207d0a2-2213-45a0-9fdf-c4666729bea7","Type":"ContainerDied","Data":"4e1ad9fef77c722cd209f73ba0819819450a0ad6cf4aae942b91e6aae79e7776"}
Oct 29 23:38:11.304038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:11.304038    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"
Oct 29 23:38:11.304277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:11.304061    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1ad9fef77c722cd209f73ba0819819450a0ad6cf4aae942b91e6aae79e7776"
Oct 29 23:38:17.463718 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.463675    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"]
Oct 29 23:38:17.464332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.464306    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2207d0a2-2213-45a0-9fdf-c4666729bea7" containerName="revoke-keys"
Oct 29 23:38:17.464332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.464331    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2207d0a2-2213-45a0-9fdf-c4666729bea7" containerName="revoke-keys"
Oct 29 23:38:17.464516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.464475    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2207d0a2-2213-45a0-9fdf-c4666729bea7" containerName="revoke-keys"
Oct 29 23:38:17.468334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.468311    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.480660 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.480628    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.480853 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.480830    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmpf\" (UniqueName: \"kubernetes.io/projected/3d1b2762-bd22-4252-8c76-af6004a0b947-kube-api-access-ptmpf\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.481016 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.480891    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.489006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.488980    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"]
Oct 29 23:38:17.500216 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.500190    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:38:17.500418 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.500214    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:38:17.500418 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.500279    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:38:17.500418 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.500289    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:38:17.582060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.582006    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmpf\" (UniqueName: \"kubernetes.io/projected/3d1b2762-bd22-4252-8c76-af6004a0b947-kube-api-access-ptmpf\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.582208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.582170    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.582316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.582304    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.582873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.582848    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.584606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.584582    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.607886 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.607839    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmpf\" (UniqueName: \"kubernetes.io/projected/3d1b2762-bd22-4252-8c76-af6004a0b947-kube-api-access-ptmpf\") pod \"maas-api-revoke-sub-d3fba056-sqv5r\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:17.789939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:17.789814    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:18.093583 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:18.093546    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"]
Oct 29 23:38:18.094529 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:38:18.094504    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1b2762_bd22_4252_8c76_af6004a0b947.slice/crio-d95fa9f5d9ac8a375440d38d27c3f80cd7a894d0a886cc1721c10cb9811d4743 WatchSource:0}: Error finding container d95fa9f5d9ac8a375440d38d27c3f80cd7a894d0a886cc1721c10cb9811d4743: Status 404 returned error can't find the container with id d95fa9f5d9ac8a375440d38d27c3f80cd7a894d0a886cc1721c10cb9811d4743
Oct 29 23:38:18.349392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:18.349327    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r" event={"ID":"3d1b2762-bd22-4252-8c76-af6004a0b947","Type":"ContainerStarted","Data":"55d5b10076ee144249b942e02d06fa8a30aaed35e0d812296daf10544697d7db"}
Oct 29 23:38:18.349392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:18.349384    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r" event={"ID":"3d1b2762-bd22-4252-8c76-af6004a0b947","Type":"ContainerStarted","Data":"d95fa9f5d9ac8a375440d38d27c3f80cd7a894d0a886cc1721c10cb9811d4743"}
Oct 29 23:38:18.398382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:18.398327    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r" podStartSLOduration=1.398314988 podStartE2EDuration="1.398314988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:18.396186833 +0000 UTC m=+1476.500447218" watchObservedRunningTime="2026-10-29 23:38:18.398314988 +0000 UTC m=+1476.502575373"
Oct 29 23:38:19.358999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:19.358869    6203 generic.go:356] "Generic (PLEG): container finished" podID="3d1b2762-bd22-4252-8c76-af6004a0b947" containerID="55d5b10076ee144249b942e02d06fa8a30aaed35e0d812296daf10544697d7db" exitCode=0
Oct 29 23:38:19.358999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:19.358951    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r" event={"ID":"3d1b2762-bd22-4252-8c76-af6004a0b947","Type":"ContainerDied","Data":"55d5b10076ee144249b942e02d06fa8a30aaed35e0d812296daf10544697d7db"}
Oct 29 23:38:20.813411 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:20.813381    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:20.918500 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:20.918463    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-xks-ca\") pod \"3d1b2762-bd22-4252-8c76-af6004a0b947\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") "
Oct 29 23:38:20.918654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:20.918566    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1b2762-bd22-4252-8c76-af6004a0b947-kube-api-access-ptmpf\" (UniqueName: \"kubernetes.io/projected/3d1b2762-bd22-4252-8c76-af6004a0b947-kube-api-access-ptmpf\") pod \"3d1b2762-bd22-4252-8c76-af6004a0b947\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") "
Oct 29 23:38:20.918702 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:20.918685    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-openshift-ca\") pod \"3d1b2762-bd22-4252-8c76-af6004a0b947\" (UID: \"3d1b2762-bd22-4252-8c76-af6004a0b947\") "
Oct 29 23:38:20.919118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:20.919083    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-openshift-ca" pod "3d1b2762-bd22-4252-8c76-af6004a0b947" (UID: "3d1b2762-bd22-4252-8c76-af6004a0b947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:38:20.920774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:20.920741    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-xks-ca" pod "3d1b2762-bd22-4252-8c76-af6004a0b947" (UID: "3d1b2762-bd22-4252-8c76-af6004a0b947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:38:20.920774 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:20.920755    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1b2762-bd22-4252-8c76-af6004a0b947-kube-api-access-ptmpf" pod "3d1b2762-bd22-4252-8c76-af6004a0b947" (UID: "3d1b2762-bd22-4252-8c76-af6004a0b947"). InnerVolumeSpecName "kube-api-access-ptmpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:38:21.020514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.020426    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:21.020514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.020455    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1b2762-bd22-4252-8c76-af6004a0b947-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:21.020514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.020465    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmpf\" (UniqueName: \"kubernetes.io/projected/3d1b2762-bd22-4252-8c76-af6004a0b947-kube-api-access-ptmpf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:21.149363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.148995    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:38:21.149363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.149063    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:38:21.388565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.388487    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"
Oct 29 23:38:21.388565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.388523    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r" event={"ID":"3d1b2762-bd22-4252-8c76-af6004a0b947","Type":"ContainerDied","Data":"d95fa9f5d9ac8a375440d38d27c3f80cd7a894d0a886cc1721c10cb9811d4743"}
Oct 29 23:38:21.388756 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:21.388570    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d95fa9f5d9ac8a375440d38d27c3f80cd7a894d0a886cc1721c10cb9811d4743"
Oct 29 23:38:36.122856 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:36.122814    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-7bzbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:38:36.123430 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:36.122904    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-7bzbg" podUID="90102371-81a6-4419-a23e-72f0ba01c595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:38:39.922964 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:38:39.922922    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:38:40.040711 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:40.040675    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"]
Oct 29 23:38:40.055358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:40.055327    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"]
Oct 29 23:38:40.058045 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:40.058024    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-134e1509-ksbph"]
Oct 29 23:38:40.061055 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:40.061029    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf1a463-cndv4"]
Oct 29 23:38:42.025254 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.025210    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7340bf-2bf3-4cb0-a9d8-889850892bd2" path="/var/lib/kubelet/pods/ae7340bf-2bf3-4cb0-a9d8-889850892bd2/volumes"
Oct 29 23:38:42.025865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.025834    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a12187-4302-46eb-b9b5-00b18f1c8171" path="/var/lib/kubelet/pods/c0a12187-4302-46eb-b9b5-00b18f1c8171/volumes"
Oct 29 23:38:42.326769 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.326690    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"]
Oct 29 23:38:42.327135 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.327120    6203 cpu_manager.go:414] "containerMap: removing container" podUID="3d1b2762-bd22-4252-8c76-af6004a0b947" containerName="revoke-keys"
Oct 29 23:38:42.327193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.327136    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1b2762-bd22-4252-8c76-af6004a0b947" containerName="revoke-keys"
Oct 29 23:38:42.327230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.327205    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1b2762-bd22-4252-8c76-af6004a0b947" containerName="revoke-keys"
Oct 29 23:38:42.331643 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.331626    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.348961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.348930    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:38:42.351360 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.351333    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"]
Oct 29 23:38:42.363281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.363253    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:38:42.363281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.363262    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:38:42.363437 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.363262    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:38:42.371502 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.371473    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.371639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.371603    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prch\" (UniqueName: \"kubernetes.io/projected/9afa91ff-e30d-4e44-92a7-85f0af972c45-kube-api-access-9prch\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.371727 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.371712    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.472725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.472689    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.472918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.472745    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.472918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.472798    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prch\" (UniqueName: \"kubernetes.io/projected/9afa91ff-e30d-4e44-92a7-85f0af972c45-kube-api-access-9prch\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.473489 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.473462    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.475161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.475135    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.499317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.499277    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prch\" (UniqueName: \"kubernetes.io/projected/9afa91ff-e30d-4e44-92a7-85f0af972c45-kube-api-access-9prch\") pod \"maas-api-revoke-sub-ae594979-fdxbp\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.644588 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.644502    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:42.947327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:42.947298    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"]
Oct 29 23:38:42.948413 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:38:42.948385    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afa91ff_e30d_4e44_92a7_85f0af972c45.slice/crio-eb772695792e3b5b7c73346f951a67e71ddc32e82bb58e0aa79556053fde9dcc WatchSource:0}: Error finding container eb772695792e3b5b7c73346f951a67e71ddc32e82bb58e0aa79556053fde9dcc: Status 404 returned error can't find the container with id eb772695792e3b5b7c73346f951a67e71ddc32e82bb58e0aa79556053fde9dcc
Oct 29 23:38:43.559060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:43.558967    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp" event={"ID":"9afa91ff-e30d-4e44-92a7-85f0af972c45","Type":"ContainerStarted","Data":"7ee10302016a6d087b4aea1900c1ed167869b7679b4ea97c18389eb3dc844717"}
Oct 29 23:38:43.559060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:43.559032    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp" event={"ID":"9afa91ff-e30d-4e44-92a7-85f0af972c45","Type":"ContainerStarted","Data":"eb772695792e3b5b7c73346f951a67e71ddc32e82bb58e0aa79556053fde9dcc"}
Oct 29 23:38:43.607139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:43.607074    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp" podStartSLOduration=1.607055863 podStartE2EDuration="1.607055863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:38:43.603710492 +0000 UTC m=+1501.707970877" watchObservedRunningTime="2026-10-29 23:38:43.607055863 +0000 UTC m=+1501.711316251"
Oct 29 23:38:44.566501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:44.566463    6203 generic.go:356] "Generic (PLEG): container finished" podID="9afa91ff-e30d-4e44-92a7-85f0af972c45" containerID="7ee10302016a6d087b4aea1900c1ed167869b7679b4ea97c18389eb3dc844717" exitCode=0
Oct 29 23:38:44.566909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:44.566549    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp" event={"ID":"9afa91ff-e30d-4e44-92a7-85f0af972c45","Type":"ContainerDied","Data":"7ee10302016a6d087b4aea1900c1ed167869b7679b4ea97c18389eb3dc844717"}
Oct 29 23:38:45.700788 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.700759    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:45.802570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.802536    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-openshift-ca\") pod \"9afa91ff-e30d-4e44-92a7-85f0af972c45\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") "
Oct 29 23:38:45.802745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.802591    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-xks-ca\") pod \"9afa91ff-e30d-4e44-92a7-85f0af972c45\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") "
Oct 29 23:38:45.802745 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.802629    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afa91ff-e30d-4e44-92a7-85f0af972c45-kube-api-access-9prch\" (UniqueName: \"kubernetes.io/projected/9afa91ff-e30d-4e44-92a7-85f0af972c45-kube-api-access-9prch\") pod \"9afa91ff-e30d-4e44-92a7-85f0af972c45\" (UID: \"9afa91ff-e30d-4e44-92a7-85f0af972c45\") "
Oct 29 23:38:45.802961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.802930    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-openshift-ca" pod "9afa91ff-e30d-4e44-92a7-85f0af972c45" (UID: "9afa91ff-e30d-4e44-92a7-85f0af972c45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:38:45.804735 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.804704    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-xks-ca" pod "9afa91ff-e30d-4e44-92a7-85f0af972c45" (UID: "9afa91ff-e30d-4e44-92a7-85f0af972c45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:38:45.804735 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.804720    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afa91ff-e30d-4e44-92a7-85f0af972c45-kube-api-access-9prch" pod "9afa91ff-e30d-4e44-92a7-85f0af972c45" (UID: "9afa91ff-e30d-4e44-92a7-85f0af972c45"). InnerVolumeSpecName "kube-api-access-9prch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:38:45.904337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.904257    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:45.904337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.904286    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afa91ff-e30d-4e44-92a7-85f0af972c45-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:45.904337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:45.904296    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prch\" (UniqueName: \"kubernetes.io/projected/9afa91ff-e30d-4e44-92a7-85f0af972c45-kube-api-access-9prch\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:38:46.579998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:46.579966    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"
Oct 29 23:38:46.579998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:46.579992    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp" event={"ID":"9afa91ff-e30d-4e44-92a7-85f0af972c45","Type":"ContainerDied","Data":"eb772695792e3b5b7c73346f951a67e71ddc32e82bb58e0aa79556053fde9dcc"}
Oct 29 23:38:46.580217 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:46.580042    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb772695792e3b5b7c73346f951a67e71ddc32e82bb58e0aa79556053fde9dcc"
Oct 29 23:38:46.984482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:46.984449    6203 scope.go:122] "RemoveContainer" containerID="c47d82c475a84afe5a2377506d6ab7d746a9479ac292e2f79c15b8c9d94c5bc7"
Oct 29 23:38:46.993901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:46.993859    6203 scope.go:122] "RemoveContainer" containerID="8c063906ed311ff96cb517c3a787d5acd0dc291cd84c2f03b0568a0354806b67"
Oct 29 23:38:47.002053 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.002031    6203 scope.go:122] "RemoveContainer" containerID="acf81e0eed2d9be4abab88d707c9ffc558c1e2b2870e1991343a50b740005efb"
Oct 29 23:38:47.010391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.010371    6203 scope.go:122] "RemoveContainer" containerID="7518338df29d17390e10bcf05fd2be5d8ae90994c8691cd60a3f33ff3e5eff20"
Oct 29 23:38:47.018676 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.018653    6203 scope.go:122] "RemoveContainer" containerID="650217440e833c99f5d11d63b665be2e4d2c0b727213d461fd9eb78c6233eef6"
Oct 29 23:38:47.029263 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.029244    6203 scope.go:122] "RemoveContainer" containerID="73dca94782d7e4b18d63121f610b80a017e348f41ce851150f319f6bae588d74"
Oct 29 23:38:47.037151 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.037134    6203 scope.go:122] "RemoveContainer" containerID="b587ed9a4acd9f47485ab40e2511f07a742d4046fa21658a9c6b6e75a7fbe47c"
Oct 29 23:38:47.044932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.044915    6203 scope.go:122] "RemoveContainer" containerID="35dcd61b52525edc577a24f76fc58777e1e82a56f7bf6c60694106ab2b7ea363"
Oct 29 23:38:47.052669 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.052653    6203 scope.go:122] "RemoveContainer" containerID="c4ff91fe3a8590b6311b9e5001e14ccb6619f022d8862f9d48e55000045bdf4e"
Oct 29 23:38:47.060366 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.060350    6203 scope.go:122] "RemoveContainer" containerID="aa3a203d58a9e876c613a5cee01bbbd6290fe50e892a9f27c229e28781e1d206"
Oct 29 23:38:47.068019 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.067990    6203 scope.go:122] "RemoveContainer" containerID="45dbc5be58d003d1b3060492cc322435ac8ac4729a3c1642a6dc28261cccb8ff"
Oct 29 23:38:47.075447 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.075425    6203 scope.go:122] "RemoveContainer" containerID="da46d465b60ce6c6c25ceb0ce1d850ab540ddedb72f86d5cf72da91da9f6113e"
Oct 29 23:38:47.082928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.082911    6203 scope.go:122] "RemoveContainer" containerID="e95b97c9dd0972d1c27b4b5211fcd86bc03485ae91c47f823963de731650634e"
Oct 29 23:38:47.090368 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:38:47.090351    6203 scope.go:122] "RemoveContainer" containerID="a829b8351cb57c28f96de67267df2121553b1da75295aed68ae118f19e48b6e5"
Oct 29 23:39:03.826833 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.826796    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"]
Oct 29 23:39:03.827423 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.827407    6203 cpu_manager.go:414] "containerMap: removing container" podUID="9afa91ff-e30d-4e44-92a7-85f0af972c45" containerName="revoke-keys"
Oct 29 23:39:03.827479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.827426    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afa91ff-e30d-4e44-92a7-85f0af972c45" containerName="revoke-keys"
Oct 29 23:39:03.827535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.827524    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afa91ff-e30d-4e44-92a7-85f0af972c45" containerName="revoke-keys"
Oct 29 23:39:03.830780 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.830761    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:03.856521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.856205    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:39:03.859358 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.859326    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"]
Oct 29 23:39:03.867836 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.867801    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:39:03.867836 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.867833    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:39:03.870862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.870834    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:39:03.904346 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.904308    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:03.904523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.904421    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpnl\" (UniqueName: \"kubernetes.io/projected/d58edd9f-3da2-438e-9862-1d3982de629e-kube-api-access-jcpnl\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:03.904523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:03.904484    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.005523 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.005485    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.005728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.005534    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpnl\" (UniqueName: \"kubernetes.io/projected/d58edd9f-3da2-438e-9862-1d3982de629e-kube-api-access-jcpnl\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.005728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.005566    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.006317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.006290    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.008018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.007995    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.032363 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.032328    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpnl\" (UniqueName: \"kubernetes.io/projected/d58edd9f-3da2-438e-9862-1d3982de629e-kube-api-access-jcpnl\") pod \"maas-api-revoke-sub-a4aef7d7-nl2dj\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.156694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.156569    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:04.462278 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:39:04.462241    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58edd9f_3da2_438e_9862_1d3982de629e.slice/crio-9f7e952e3f08a9a7591111e551872384e390ec422934fc5ee5a313ca1722a0c3 WatchSource:0}: Error finding container 9f7e952e3f08a9a7591111e551872384e390ec422934fc5ee5a313ca1722a0c3: Status 404 returned error can't find the container with id 9f7e952e3f08a9a7591111e551872384e390ec422934fc5ee5a313ca1722a0c3
Oct 29 23:39:04.462910 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.462871    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"]
Oct 29 23:39:04.715985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:04.715873    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj" event={"ID":"d58edd9f-3da2-438e-9862-1d3982de629e","Type":"ContainerStarted","Data":"9f7e952e3f08a9a7591111e551872384e390ec422934fc5ee5a313ca1722a0c3"}
Oct 29 23:39:05.722567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:05.722532    6203 generic.go:356] "Generic (PLEG): container finished" podID="d58edd9f-3da2-438e-9862-1d3982de629e" containerID="28f9e183e724141a23c2a48f86a2c111a21e67fee3b95aa31eede635cf50836b" exitCode=0
Oct 29 23:39:05.722979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:05.722580    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj" event={"ID":"d58edd9f-3da2-438e-9862-1d3982de629e","Type":"ContainerDied","Data":"28f9e183e724141a23c2a48f86a2c111a21e67fee3b95aa31eede635cf50836b"}
Oct 29 23:39:06.046299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.046206    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"]
Oct 29 23:39:06.051632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.051601    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5584c3af-6b4dc"]
Oct 29 23:39:06.860116 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.860083    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:06.935291 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.935261    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58edd9f-3da2-438e-9862-1d3982de629e-kube-api-access-jcpnl\" (UniqueName: \"kubernetes.io/projected/d58edd9f-3da2-438e-9862-1d3982de629e-kube-api-access-jcpnl\") pod \"d58edd9f-3da2-438e-9862-1d3982de629e\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") "
Oct 29 23:39:06.935494 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.935313    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-xks-ca\") pod \"d58edd9f-3da2-438e-9862-1d3982de629e\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") "
Oct 29 23:39:06.935494 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.935442    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-openshift-ca\") pod \"d58edd9f-3da2-438e-9862-1d3982de629e\" (UID: \"d58edd9f-3da2-438e-9862-1d3982de629e\") "
Oct 29 23:39:06.935787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.935762    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-openshift-ca" pod "d58edd9f-3da2-438e-9862-1d3982de629e" (UID: "d58edd9f-3da2-438e-9862-1d3982de629e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:39:06.937822 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.937788    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-xks-ca" pod "d58edd9f-3da2-438e-9862-1d3982de629e" (UID: "d58edd9f-3da2-438e-9862-1d3982de629e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:39:06.938103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:06.938078    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58edd9f-3da2-438e-9862-1d3982de629e-kube-api-access-jcpnl" pod "d58edd9f-3da2-438e-9862-1d3982de629e" (UID: "d58edd9f-3da2-438e-9862-1d3982de629e"). InnerVolumeSpecName "kube-api-access-jcpnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:39:07.036507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:07.036414    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:07.036507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:07.036448    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58edd9f-3da2-438e-9862-1d3982de629e-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:07.036507 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:07.036462    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcpnl\" (UniqueName: \"kubernetes.io/projected/d58edd9f-3da2-438e-9862-1d3982de629e-kube-api-access-jcpnl\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:07.735030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:07.734996    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"
Oct 29 23:39:07.735030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:07.735013    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj" event={"ID":"d58edd9f-3da2-438e-9862-1d3982de629e","Type":"ContainerDied","Data":"9f7e952e3f08a9a7591111e551872384e390ec422934fc5ee5a313ca1722a0c3"}
Oct 29 23:39:07.735256 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:07.735061    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7e952e3f08a9a7591111e551872384e390ec422934fc5ee5a313ca1722a0c3"
Oct 29 23:39:08.017012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:08.016917    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd" path="/var/lib/kubelet/pods/fff5ac49-1de5-4e36-abe0-ee91b4ce1ebd/volumes"
Oct 29 23:39:09.925025 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:39:09.924986    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:39:17.031304 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:17.031266    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"]
Oct 29 23:39:17.046004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:17.045973    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cee766-q6n2f"]
Oct 29 23:39:18.018013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:18.017976    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6f0891-7dae-4f01-a7b8-cd872705d6f1" path="/var/lib/kubelet/pods/2b6f0891-7dae-4f01-a7b8-cd872705d6f1/volumes"
Oct 29 23:39:27.097486 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.097408    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"]
Oct 29 23:39:27.097855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.097813    6203 cpu_manager.go:414] "containerMap: removing container" podUID="d58edd9f-3da2-438e-9862-1d3982de629e" containerName="revoke-keys"
Oct 29 23:39:27.097855 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.097824    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58edd9f-3da2-438e-9862-1d3982de629e" containerName="revoke-keys"
Oct 29 23:39:27.097944 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.097897    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58edd9f-3da2-438e-9862-1d3982de629e" containerName="revoke-keys"
Oct 29 23:39:27.102612 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.102587    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.123699 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.123670    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"]
Oct 29 23:39:27.135707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.135673    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:39:27.135707 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.135680    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:39:27.135945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.135708    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:39:27.135945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.135749    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:39:27.147274 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.147252    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.147373 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.147291    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nlf6\" (UniqueName: \"kubernetes.io/projected/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-kube-api-access-9nlf6\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.147373 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.147319    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.248639 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.248601    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.248799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.248650    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nlf6\" (UniqueName: \"kubernetes.io/projected/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-kube-api-access-9nlf6\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.248799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.248671    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.249357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.249327    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.251059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.251039    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.277260 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.277234    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nlf6\" (UniqueName: \"kubernetes.io/projected/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-kube-api-access-9nlf6\") pod \"maas-api-revoke-sub-79805ca9-dms6z\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.415041 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.414958    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:27.718892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.718850    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"]
Oct 29 23:39:27.720425 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:39:27.720393    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc4bbe6_ff64_4e2b_a8e6_fa8fa9f6aa87.slice/crio-5285eae54aa56e1b0680d00c4df0ce56a1da967e7c3664a918794642c757cb56 WatchSource:0}: Error finding container 5285eae54aa56e1b0680d00c4df0ce56a1da967e7c3664a918794642c757cb56: Status 404 returned error can't find the container with id 5285eae54aa56e1b0680d00c4df0ce56a1da967e7c3664a918794642c757cb56
Oct 29 23:39:27.861604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:27.861571    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z" event={"ID":"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87","Type":"ContainerStarted","Data":"5285eae54aa56e1b0680d00c4df0ce56a1da967e7c3664a918794642c757cb56"}
Oct 29 23:39:28.869098 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:28.869008    6203 generic.go:356] "Generic (PLEG): container finished" podID="3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" containerID="6187036548c0df90a273efbb307465e3cdecfc380bffbbc9bf682b6521944eb4" exitCode=0
Oct 29 23:39:28.869475 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:28.869095    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z" event={"ID":"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87","Type":"ContainerDied","Data":"6187036548c0df90a273efbb307465e3cdecfc380bffbbc9bf682b6521944eb4"}
Oct 29 23:39:30.010471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.010449    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:30.076793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.076754    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-xks-ca\") pod \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") "
Oct 29 23:39:30.077009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.076826    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-openshift-ca\") pod \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") "
Oct 29 23:39:30.077009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.076948    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-kube-api-access-9nlf6\" (UniqueName: \"kubernetes.io/projected/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-kube-api-access-9nlf6\") pod \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\" (UID: \"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87\") "
Oct 29 23:39:30.077267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.077239    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-openshift-ca" pod "3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" (UID: "3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:39:30.077739 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.077457    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:30.079449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.079425    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-xks-ca" pod "3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" (UID: "3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:39:30.079535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.079464    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-kube-api-access-9nlf6" pod "3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" (UID: "3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87"). InnerVolumeSpecName "kube-api-access-9nlf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:39:30.178566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.178534    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:30.178566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.178565    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nlf6\" (UniqueName: \"kubernetes.io/projected/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87-kube-api-access-9nlf6\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:30.884609 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.884579    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"
Oct 29 23:39:30.884790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.884569    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z" event={"ID":"3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87","Type":"ContainerDied","Data":"5285eae54aa56e1b0680d00c4df0ce56a1da967e7c3664a918794642c757cb56"}
Oct 29 23:39:30.884790 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:30.884708    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5285eae54aa56e1b0680d00c4df0ce56a1da967e7c3664a918794642c757cb56"
Oct 29 23:39:36.036448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:36.036412    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"]
Oct 29 23:39:36.050934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:36.050870    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"]
Oct 29 23:39:36.052851 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:36.052828    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2d22e6e-98rdt"]
Oct 29 23:39:36.054431 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:36.054413    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bab26dc-5dxkp"]
Oct 29 23:39:38.017850 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:38.017818    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293eae7c-c172-4580-b4d4-5933e97ed0fc" path="/var/lib/kubelet/pods/293eae7c-c172-4580-b4d4-5933e97ed0fc/volumes"
Oct 29 23:39:38.018435 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:38.018419    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b817b21-4bf2-415a-928b-cb2f20a5b995" path="/var/lib/kubelet/pods/8b817b21-4bf2-415a-928b-cb2f20a5b995/volumes"
Oct 29 23:39:38.029161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:38.029137    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"]
Oct 29 23:39:38.043248 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:38.043223    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde306b2-4w6lt"]
Oct 29 23:39:39.935448 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:39:39.935403    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:39:40.019938 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:40.019900    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2156d51e-1686-4d52-9c06-3aee9f8e15f6" path="/var/lib/kubelet/pods/2156d51e-1686-4d52-9c06-3aee9f8e15f6/volumes"
Oct 29 23:39:43.900658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.900617    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"]
Oct 29 23:39:43.901103 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.901086    6203 cpu_manager.go:414] "containerMap: removing container" podUID="3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" containerName="revoke-keys"
Oct 29 23:39:43.901153 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.901105    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" containerName="revoke-keys"
Oct 29 23:39:43.901188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.901182    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" containerName="revoke-keys"
Oct 29 23:39:43.905703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.905685    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:43.923231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.923202    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:39:43.925263 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.925238    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"]
Oct 29 23:39:43.933526 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.933503    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:43.933641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.933534    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:43.933703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.933682    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ckf\" (UniqueName: \"kubernetes.io/projected/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-kube-api-access-97ckf\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:43.937380 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.937363    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:39:43.937773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.937737    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:39:43.937933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:43.937807    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:39:44.034840 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.034808    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:44.035026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.034923    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ckf\" (UniqueName: \"kubernetes.io/projected/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-kube-api-access-97ckf\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:44.035026 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.034963    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:44.035584 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.035561    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:44.037082 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.037055    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:44.061525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.061494    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ckf\" (UniqueName: \"kubernetes.io/projected/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-kube-api-access-97ckf\") pod \"maas-api-revoke-sub-0b7e8edb-p9wdh\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:44.218445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.218411    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:44.521480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:44.521450    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"]
Oct 29 23:39:44.521799 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:39:44.521770    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2467e88d_1bb3_4840_bf1a_a7c16976eaa9.slice/crio-8412f1d7bf2018c02436edfed95bbf02f81b2b3f62caddcb5599d7158d652768 WatchSource:0}: Error finding container 8412f1d7bf2018c02436edfed95bbf02f81b2b3f62caddcb5599d7158d652768: Status 404 returned error can't find the container with id 8412f1d7bf2018c02436edfed95bbf02f81b2b3f62caddcb5599d7158d652768
Oct 29 23:39:45.006889 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:45.006847    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh" event={"ID":"2467e88d-1bb3-4840-bf1a-a7c16976eaa9","Type":"ContainerStarted","Data":"cd25a2ee5efa07a0e70a32d3650e4d8ff49476096837334b4791ad79644a609a"}
Oct 29 23:39:45.007288 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:45.006932    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh" event={"ID":"2467e88d-1bb3-4840-bf1a-a7c16976eaa9","Type":"ContainerStarted","Data":"8412f1d7bf2018c02436edfed95bbf02f81b2b3f62caddcb5599d7158d652768"}
Oct 29 23:39:45.053276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:45.053222    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh" podStartSLOduration=2.053208329 podStartE2EDuration="2.053208329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:39:45.052688057 +0000 UTC m=+1563.156948477" watchObservedRunningTime="2026-10-29 23:39:45.053208329 +0000 UTC m=+1563.157468709"
Oct 29 23:39:46.014687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:46.014653    6203 generic.go:356] "Generic (PLEG): container finished" podID="2467e88d-1bb3-4840-bf1a-a7c16976eaa9" containerID="cd25a2ee5efa07a0e70a32d3650e4d8ff49476096837334b4791ad79644a609a" exitCode=0
Oct 29 23:39:46.017075 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:46.017050    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh" event={"ID":"2467e88d-1bb3-4840-bf1a-a7c16976eaa9","Type":"ContainerDied","Data":"cd25a2ee5efa07a0e70a32d3650e4d8ff49476096837334b4791ad79644a609a"}
Oct 29 23:39:47.153620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.153595    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:47.170112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.170078    6203 scope.go:122] "RemoveContainer" containerID="5afb081f6d07f156917cc6ab7b9def4815fe5ec160288829a0020f72dfaacae2"
Oct 29 23:39:47.179124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.179108    6203 scope.go:122] "RemoveContainer" containerID="b2894d6f4141690646ae5d59b516ceb2a7ab389e19e6b7672ab7cd50d275da39"
Oct 29 23:39:47.187177 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.187145    6203 scope.go:122] "RemoveContainer" containerID="53419509ea6c226823a6fd2adb96cd6019817f07e752483404ae9b87cc8622cb"
Oct 29 23:39:47.195176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.195158    6203 scope.go:122] "RemoveContainer" containerID="e71a7c4368800da2305e7aad0662b40b33f042af8c58285963ec0d08d3ac6c9c"
Oct 29 23:39:47.204979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.204927    6203 scope.go:122] "RemoveContainer" containerID="5df0451536e15ba5693ac0d049cd88de2b8c7009bc35fb5bdcb07d7bac844797"
Oct 29 23:39:47.214556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.214531    6203 scope.go:122] "RemoveContainer" containerID="94d1c6c8cca2336d8bb88827e6e4361513933e0d9256a28449e897745e9c63c3"
Oct 29 23:39:47.225654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.225632    6203 scope.go:122] "RemoveContainer" containerID="afcce26e34a8937eb366685c8a6b4f2923f4d463d7482a7b6863e823a8ae8df0"
Oct 29 23:39:47.266503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.266484    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-openshift-ca\") pod \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") "
Oct 29 23:39:47.266575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.266520    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-kube-api-access-97ckf\" (UniqueName: \"kubernetes.io/projected/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-kube-api-access-97ckf\") pod \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") "
Oct 29 23:39:47.266620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.266575    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-xks-ca\") pod \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\" (UID: \"2467e88d-1bb3-4840-bf1a-a7c16976eaa9\") "
Oct 29 23:39:47.266829 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.266805    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-openshift-ca" pod "2467e88d-1bb3-4840-bf1a-a7c16976eaa9" (UID: "2467e88d-1bb3-4840-bf1a-a7c16976eaa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:39:47.268889 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.268851    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-xks-ca" pod "2467e88d-1bb3-4840-bf1a-a7c16976eaa9" (UID: "2467e88d-1bb3-4840-bf1a-a7c16976eaa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:39:47.268988 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.268950    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-kube-api-access-97ckf" pod "2467e88d-1bb3-4840-bf1a-a7c16976eaa9" (UID: "2467e88d-1bb3-4840-bf1a-a7c16976eaa9"). InnerVolumeSpecName "kube-api-access-97ckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:39:47.368036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.368001    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:47.368036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.368027    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ckf\" (UniqueName: \"kubernetes.io/projected/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-kube-api-access-97ckf\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:47.368036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:47.368042    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2467e88d-1bb3-4840-bf1a-a7c16976eaa9-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:48.027418 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:48.027390    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh" event={"ID":"2467e88d-1bb3-4840-bf1a-a7c16976eaa9","Type":"ContainerDied","Data":"8412f1d7bf2018c02436edfed95bbf02f81b2b3f62caddcb5599d7158d652768"}
Oct 29 23:39:48.027418 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:48.027410    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"
Oct 29 23:39:48.027620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:48.027441    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8412f1d7bf2018c02436edfed95bbf02f81b2b3f62caddcb5599d7158d652768"
Oct 29 23:39:53.030546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:53.030511    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"]
Oct 29 23:39:53.045125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:53.045097    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54e704d6-zfv7r"]
Oct 29 23:39:54.017597 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.017563    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb491355-dec4-44b0-8c70-cbcadade530f" path="/var/lib/kubelet/pods/eb491355-dec4-44b0-8c70-cbcadade530f/volumes"
Oct 29 23:39:54.634075 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.634038    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"]
Oct 29 23:39:54.634648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.634627    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2467e88d-1bb3-4840-bf1a-a7c16976eaa9" containerName="revoke-keys"
Oct 29 23:39:54.634740 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.634651    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2467e88d-1bb3-4840-bf1a-a7c16976eaa9" containerName="revoke-keys"
Oct 29 23:39:54.634800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.634778    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2467e88d-1bb3-4840-bf1a-a7c16976eaa9" containerName="revoke-keys"
Oct 29 23:39:54.640260 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.640236    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.658274 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.658249    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"]
Oct 29 23:39:54.670374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.670348    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:39:54.670519 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.670393    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:39:54.736987 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.736938    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.737161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.737132    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xkl\" (UniqueName: \"kubernetes.io/projected/7f05208d-8209-433d-9a87-5ef76ef5fd2b-kube-api-access-t6xkl\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.737203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.737171    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.748561 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.748543    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:39:54.748627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.748562    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:39:54.838023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.837981    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xkl\" (UniqueName: \"kubernetes.io/projected/7f05208d-8209-433d-9a87-5ef76ef5fd2b-kube-api-access-t6xkl\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.838228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.838029    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.838228 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.838179    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.839112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.839069    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.840412 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.840393    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.864171 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.864133    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xkl\" (UniqueName: \"kubernetes.io/projected/7f05208d-8209-433d-9a87-5ef76ef5fd2b-kube-api-access-t6xkl\") pod \"maas-api-revoke-sub-3d4156f1-l9q27\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:54.952731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:54.952697    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:55.251324 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:55.251297    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"]
Oct 29 23:39:55.252399 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:39:55.252365    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f05208d_8209_433d_9a87_5ef76ef5fd2b.slice/crio-7ec6fbd6933120e6a5e5730a510c654cc499518d3d58971e2f234ead58eb6f9f WatchSource:0}: Error finding container 7ec6fbd6933120e6a5e5730a510c654cc499518d3d58971e2f234ead58eb6f9f: Status 404 returned error can't find the container with id 7ec6fbd6933120e6a5e5730a510c654cc499518d3d58971e2f234ead58eb6f9f
Oct 29 23:39:56.072296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:56.072262    6203 generic.go:356] "Generic (PLEG): container finished" podID="7f05208d-8209-433d-9a87-5ef76ef5fd2b" containerID="19a15c631e863e17c7f8d0f48b8b98ce9dfce588e923d90730808dd966b87a04" exitCode=0
Oct 29 23:39:56.072703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:56.072351    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27" event={"ID":"7f05208d-8209-433d-9a87-5ef76ef5fd2b","Type":"ContainerDied","Data":"19a15c631e863e17c7f8d0f48b8b98ce9dfce588e923d90730808dd966b87a04"}
Oct 29 23:39:56.072703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:56.072408    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27" event={"ID":"7f05208d-8209-433d-9a87-5ef76ef5fd2b","Type":"ContainerStarted","Data":"7ec6fbd6933120e6a5e5730a510c654cc499518d3d58971e2f234ead58eb6f9f"}
Oct 29 23:39:57.211594 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.211568    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:57.362024 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.361936    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f05208d-8209-433d-9a87-5ef76ef5fd2b-kube-api-access-t6xkl\" (UniqueName: \"kubernetes.io/projected/7f05208d-8209-433d-9a87-5ef76ef5fd2b-kube-api-access-t6xkl\") pod \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") "
Oct 29 23:39:57.362024 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.361979    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-openshift-ca\") pod \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") "
Oct 29 23:39:57.362332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.362033    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-xks-ca\") pod \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\" (UID: \"7f05208d-8209-433d-9a87-5ef76ef5fd2b\") "
Oct 29 23:39:57.362446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.362422    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-openshift-ca" pod "7f05208d-8209-433d-9a87-5ef76ef5fd2b" (UID: "7f05208d-8209-433d-9a87-5ef76ef5fd2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:39:57.364226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.364194    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f05208d-8209-433d-9a87-5ef76ef5fd2b-kube-api-access-t6xkl" pod "7f05208d-8209-433d-9a87-5ef76ef5fd2b" (UID: "7f05208d-8209-433d-9a87-5ef76ef5fd2b"). InnerVolumeSpecName "kube-api-access-t6xkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:39:57.364226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.364208    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-xks-ca" pod "7f05208d-8209-433d-9a87-5ef76ef5fd2b" (UID: "7f05208d-8209-433d-9a87-5ef76ef5fd2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:39:57.463276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.463237    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6xkl\" (UniqueName: \"kubernetes.io/projected/7f05208d-8209-433d-9a87-5ef76ef5fd2b-kube-api-access-t6xkl\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:57.463276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.463269    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:57.463276 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:57.463281    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f05208d-8209-433d-9a87-5ef76ef5fd2b-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:39:58.084503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:58.084475    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"
Oct 29 23:39:58.084503 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:58.084500    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27" event={"ID":"7f05208d-8209-433d-9a87-5ef76ef5fd2b","Type":"ContainerDied","Data":"7ec6fbd6933120e6a5e5730a510c654cc499518d3d58971e2f234ead58eb6f9f"}
Oct 29 23:39:58.084733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:39:58.084554    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ec6fbd6933120e6a5e5730a510c654cc499518d3d58971e2f234ead58eb6f9f"
Oct 29 23:40:02.030182 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:02.030146    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"]
Oct 29 23:40:02.045120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:02.045068    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"]
Oct 29 23:40:02.047001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:02.046964    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf0887-mzlf6"]
Oct 29 23:40:02.048608 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:02.048578    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee104db0-d55pz"]
Oct 29 23:40:04.023370 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:04.023325    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33628ce-551c-4a00-bfc4-083a89ad8da1" path="/var/lib/kubelet/pods/b33628ce-551c-4a00-bfc4-083a89ad8da1/volumes"
Oct 29 23:40:04.023960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:04.023941    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d50f3d-7e8a-4a14-9864-3d9525668f82" path="/var/lib/kubelet/pods/d4d50f3d-7e8a-4a14-9864-3d9525668f82/volumes"
Oct 29 23:40:08.032517 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:08.032481    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"]
Oct 29 23:40:08.047102 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:08.047065    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"]
Oct 29 23:40:08.049896 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:08.049848    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd0cd8a-6ddwd"]
Oct 29 23:40:08.052414 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:08.052389    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fde25c-cd8dd"]
Oct 29 23:40:09.943954 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:40:09.943908    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:40:10.020782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:10.020743    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4f0854-95f0-4f11-9419-b5cefd13a9c0" path="/var/lib/kubelet/pods/4b4f0854-95f0-4f11-9419-b5cefd13a9c0/volumes"
Oct 29 23:40:10.021375 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:10.021353    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9805adb0-7e4c-4667-95f8-05754ba79910" path="/var/lib/kubelet/pods/9805adb0-7e4c-4667-95f8-05754ba79910/volumes"
Oct 29 23:40:15.926612 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.926583    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"]
Oct 29 23:40:15.940661 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.940633    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l9q27"]
Oct 29 23:40:15.974031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.973997    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"]
Oct 29 23:40:15.974389 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.974377    6203 cpu_manager.go:414] "containerMap: removing container" podUID="7f05208d-8209-433d-9a87-5ef76ef5fd2b" containerName="revoke-keys"
Oct 29 23:40:15.974445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.974391    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f05208d-8209-433d-9a87-5ef76ef5fd2b" containerName="revoke-keys"
Oct 29 23:40:15.974482 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.974468    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f05208d-8209-433d-9a87-5ef76ef5fd2b" containerName="revoke-keys"
Oct 29 23:40:15.979008 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.978984    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:15.996291 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.996263    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:40:15.996453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.996298    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:40:15.996453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.996383    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:40:15.996585 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:15.996452    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:40:16.006227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.003254    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"]
Oct 29 23:40:16.017391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.017360    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f05208d-8209-433d-9a87-5ef76ef5fd2b" path="/var/lib/kubelet/pods/7f05208d-8209-433d-9a87-5ef76ef5fd2b/volumes"
Oct 29 23:40:16.061214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.061161    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.061434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.061287    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmr5\" (UniqueName: \"kubernetes.io/projected/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-kube-api-access-9pmr5\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.061434 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.061339    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.162596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.162544    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmr5\" (UniqueName: \"kubernetes.io/projected/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-kube-api-access-9pmr5\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.162596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.162611    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.162840 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.162736    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.163418 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.163390    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.165066 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.165042    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.188934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.188846    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmr5\" (UniqueName: \"kubernetes.io/projected/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-kube-api-access-9pmr5\") pod \"maas-api-revoke-sub-3d4156f1-7dfs2\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.291959 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.291926    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:16.596431 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:16.596401    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"]
Oct 29 23:40:16.598059 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:40:16.598032    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2686ae_5ac5_4b10_8f45_0488f8aaf649.slice/crio-4afe6b6da5786ac06cd543a86f1c75eb30963d92a3a2a4ef01819a60acba26fa WatchSource:0}: Error finding container 4afe6b6da5786ac06cd543a86f1c75eb30963d92a3a2a4ef01819a60acba26fa: Status 404 returned error can't find the container with id 4afe6b6da5786ac06cd543a86f1c75eb30963d92a3a2a4ef01819a60acba26fa
Oct 29 23:40:17.208575 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:17.208539    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2" event={"ID":"2f2686ae-5ac5-4b10-8f45-0488f8aaf649","Type":"ContainerStarted","Data":"b6562b0a077f22112cea0dd8b43c6ab5f274d5ea1a473eb3469b3190ff54fb1b"}
Oct 29 23:40:17.209092 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:17.208591    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2" event={"ID":"2f2686ae-5ac5-4b10-8f45-0488f8aaf649","Type":"ContainerStarted","Data":"4afe6b6da5786ac06cd543a86f1c75eb30963d92a3a2a4ef01819a60acba26fa"}
Oct 29 23:40:17.255952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:17.255897    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2" podStartSLOduration=2.255868866 podStartE2EDuration="2.255868866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:40:17.253631549 +0000 UTC m=+1595.357891947" watchObservedRunningTime="2026-10-29 23:40:17.255868866 +0000 UTC m=+1595.360129255"
Oct 29 23:40:18.218719 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:18.218683    6203 generic.go:356] "Generic (PLEG): container finished" podID="2f2686ae-5ac5-4b10-8f45-0488f8aaf649" containerID="b6562b0a077f22112cea0dd8b43c6ab5f274d5ea1a473eb3469b3190ff54fb1b" exitCode=0
Oct 29 23:40:18.219149 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:18.218773    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2" event={"ID":"2f2686ae-5ac5-4b10-8f45-0488f8aaf649","Type":"ContainerDied","Data":"b6562b0a077f22112cea0dd8b43c6ab5f274d5ea1a473eb3469b3190ff54fb1b"}
Oct 29 23:40:19.356946 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.356920    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:19.395160 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.395128    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-kube-api-access-9pmr5\" (UniqueName: \"kubernetes.io/projected/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-kube-api-access-9pmr5\") pod \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") "
Oct 29 23:40:19.395335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.395193    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-xks-ca\") pod \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") "
Oct 29 23:40:19.395335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.395249    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-openshift-ca\") pod \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\" (UID: \"2f2686ae-5ac5-4b10-8f45-0488f8aaf649\") "
Oct 29 23:40:19.395723 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.395690    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-openshift-ca" pod "2f2686ae-5ac5-4b10-8f45-0488f8aaf649" (UID: "2f2686ae-5ac5-4b10-8f45-0488f8aaf649"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:40:19.397223 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.397196    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-kube-api-access-9pmr5" pod "2f2686ae-5ac5-4b10-8f45-0488f8aaf649" (UID: "2f2686ae-5ac5-4b10-8f45-0488f8aaf649"). InnerVolumeSpecName "kube-api-access-9pmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:40:19.397327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.397299    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-xks-ca" pod "2f2686ae-5ac5-4b10-8f45-0488f8aaf649" (UID: "2f2686ae-5ac5-4b10-8f45-0488f8aaf649"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:40:19.496606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.496527    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmr5\" (UniqueName: \"kubernetes.io/projected/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-kube-api-access-9pmr5\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:19.496606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.496559    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:19.496606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:19.496569    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f2686ae-5ac5-4b10-8f45-0488f8aaf649-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:20.233079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:20.233046    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"
Oct 29 23:40:20.233282 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:20.233082    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2" event={"ID":"2f2686ae-5ac5-4b10-8f45-0488f8aaf649","Type":"ContainerDied","Data":"4afe6b6da5786ac06cd543a86f1c75eb30963d92a3a2a4ef01819a60acba26fa"}
Oct 29 23:40:20.233282 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:20.233128    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afe6b6da5786ac06cd543a86f1c75eb30963d92a3a2a4ef01819a60acba26fa"
Oct 29 23:40:22.205435 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.205394    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"]
Oct 29 23:40:22.205846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.205616    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh" podUID="def45b17-34b8-4375-aae1-079d041c2b07" containerName="discovery" containerID="cri-o://9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887" gracePeriod=30
Oct 29 23:40:22.464654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.464592    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:40:22.525919 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.525857    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-cacerts\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-cacerts\") pod \"def45b17-34b8-4375-aae1-079d041c2b07\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") "
Oct 29 23:40:22.526296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.525948    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-kubeconfig\") pod \"def45b17-34b8-4375-aae1-079d041c2b07\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") "
Oct 29 23:40:22.526296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.525980    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-kube-api-access-n9msv\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-kube-api-access-n9msv\") pod \"def45b17-34b8-4375-aae1-079d041c2b07\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") "
Oct 29 23:40:22.526296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.526068    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-dns-cert\") pod \"def45b17-34b8-4375-aae1-079d041c2b07\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") "
Oct 29 23:40:22.526296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.526120    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def45b17-34b8-4375-aae1-079d041c2b07-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/def45b17-34b8-4375-aae1-079d041c2b07-local-certs\") pod \"def45b17-34b8-4375-aae1-079d041c2b07\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") "
Oct 29 23:40:22.526296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.526161    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-istio-token\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-istio-token\") pod \"def45b17-34b8-4375-aae1-079d041c2b07\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") "
Oct 29 23:40:22.526296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.526199    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-ca-configmap\") pod \"def45b17-34b8-4375-aae1-079d041c2b07\" (UID: \"def45b17-34b8-4375-aae1-079d041c2b07\") "
Oct 29 23:40:22.526842 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.526817    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-ca-configmap" pod "def45b17-34b8-4375-aae1-079d041c2b07" (UID: "def45b17-34b8-4375-aae1-079d041c2b07"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:40:22.528802 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.528725    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-cacerts" pod "def45b17-34b8-4375-aae1-079d041c2b07" (UID: "def45b17-34b8-4375-aae1-079d041c2b07"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:40:22.528958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.528837    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-kubeconfig" pod "def45b17-34b8-4375-aae1-079d041c2b07" (UID: "def45b17-34b8-4375-aae1-079d041c2b07"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:40:22.529134 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.529102    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-istio-token" pod "def45b17-34b8-4375-aae1-079d041c2b07" (UID: "def45b17-34b8-4375-aae1-079d041c2b07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:40:22.529522 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.529499    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-dns-cert" pod "def45b17-34b8-4375-aae1-079d041c2b07" (UID: "def45b17-34b8-4375-aae1-079d041c2b07"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:40:22.530297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.530268    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-kube-api-access-n9msv" pod "def45b17-34b8-4375-aae1-079d041c2b07" (UID: "def45b17-34b8-4375-aae1-079d041c2b07"). InnerVolumeSpecName "kube-api-access-n9msv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:40:22.530955 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.530923    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def45b17-34b8-4375-aae1-079d041c2b07-local-certs" pod "def45b17-34b8-4375-aae1-079d041c2b07" (UID: "def45b17-34b8-4375-aae1-079d041c2b07"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:40:22.627335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.627303    6203 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-cacerts\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:22.627335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.627332    6203 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-kubeconfig\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:22.627557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.627348    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9msv\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-kube-api-access-n9msv\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:22.627557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.627361    6203 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-dns-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:22.627557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.627373    6203 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/def45b17-34b8-4375-aae1-079d041c2b07-local-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:22.627557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.627384    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/def45b17-34b8-4375-aae1-079d041c2b07-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:22.627557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:22.627394    6203 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/def45b17-34b8-4375-aae1-079d041c2b07-istio-csr-ca-configmap\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:23.256777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.256738    6203 generic.go:356] "Generic (PLEG): container finished" podID="def45b17-34b8-4375-aae1-079d041c2b07" containerID="9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887" exitCode=0
Oct 29 23:40:23.257267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.256792    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh" event={"ID":"def45b17-34b8-4375-aae1-079d041c2b07","Type":"ContainerDied","Data":"9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887"}
Oct 29 23:40:23.257267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.256806    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"
Oct 29 23:40:23.257267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.256841    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh" event={"ID":"def45b17-34b8-4375-aae1-079d041c2b07","Type":"ContainerDied","Data":"612f443257613a4ea7865cc6a0e387410e4cca413da5a980cd3e1d80b65b6b85"}
Oct 29 23:40:23.257267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.256868    6203 scope.go:122] "RemoveContainer" containerID="9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887"
Oct 29 23:40:23.267154 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.267136    6203 scope.go:122] "RemoveContainer" containerID="9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887"
Oct 29 23:40:23.267436 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:40:23.267418    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887\": container with ID starting with 9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887 not found: ID does not exist" containerID="9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887"
Oct 29 23:40:23.267479 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.267444    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887"} err="failed to get container status \"9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887\": rpc error: code = NotFound desc = could not find container \"9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887\": container with ID starting with 9916a15ee6fd47d1113a4d9204c61060c0d18c4edc9a8d60357853655343a887 not found: ID does not exist"
Oct 29 23:40:23.329996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.329968    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"]
Oct 29 23:40:23.344206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:23.344179    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5zsnh"]
Oct 29 23:40:24.017360 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:24.017327    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def45b17-34b8-4375-aae1-079d041c2b07" path="/var/lib/kubelet/pods/def45b17-34b8-4375-aae1-079d041c2b07/volumes"
Oct 29 23:40:28.032067 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:28.032031    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"]
Oct 29 23:40:28.046709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:28.046676    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1be84620-nkj5n"]
Oct 29 23:40:30.016818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:30.016783    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52ddb5e-d6ce-4494-9281-c7940785627f" path="/var/lib/kubelet/pods/f52ddb5e-d6ce-4494-9281-c7940785627f/volumes"
Oct 29 23:40:39.449557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.449519    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"]
Oct 29 23:40:39.452000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.451980    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7dfs2"]
Oct 29 23:40:39.500698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.500664    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"]
Oct 29 23:40:39.501109 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.501095    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2f2686ae-5ac5-4b10-8f45-0488f8aaf649" containerName="revoke-keys"
Oct 29 23:40:39.501170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.501110    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2686ae-5ac5-4b10-8f45-0488f8aaf649" containerName="revoke-keys"
Oct 29 23:40:39.501170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.501125    6203 cpu_manager.go:414] "containerMap: removing container" podUID="def45b17-34b8-4375-aae1-079d041c2b07" containerName="discovery"
Oct 29 23:40:39.501170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.501131    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def45b17-34b8-4375-aae1-079d041c2b07" containerName="discovery"
Oct 29 23:40:39.501272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.501189    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2686ae-5ac5-4b10-8f45-0488f8aaf649" containerName="revoke-keys"
Oct 29 23:40:39.501272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.501198    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="def45b17-34b8-4375-aae1-079d041c2b07" containerName="discovery"
Oct 29 23:40:39.504484 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.504468    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.522976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.522949    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:40:39.523152 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.523063    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:40:39.523259 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.523157    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:40:39.523259 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.523206    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:40:39.525257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.525237    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"]
Oct 29 23:40:39.603631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.603596    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.603821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.603688    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.603821 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.603732    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8244\" (UniqueName: \"kubernetes.io/projected/2c8b1696-7b99-49b0-b192-3576a4843ada-kube-api-access-s8244\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.704698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.704607    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8244\" (UniqueName: \"kubernetes.io/projected/2c8b1696-7b99-49b0-b192-3576a4843ada-kube-api-access-s8244\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.704698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.704679    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.704926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.704725    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.705323 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.705304    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.706994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.706969    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.731191 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.731166    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8244\" (UniqueName: \"kubernetes.io/projected/2c8b1696-7b99-49b0-b192-3576a4843ada-kube-api-access-s8244\") pod \"maas-api-revoke-sub-3d4156f1-tszfx\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.817039 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:39.817006    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:39.924755 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:40:39.924713    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:40:40.017683 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:40.017598    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2686ae-5ac5-4b10-8f45-0488f8aaf649" path="/var/lib/kubelet/pods/2f2686ae-5ac5-4b10-8f45-0488f8aaf649/volumes"
Oct 29 23:40:40.120815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:40.120788    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"]
Oct 29 23:40:40.123169 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:40:40.123138    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8b1696_7b99_49b0_b192_3576a4843ada.slice/crio-1afbabc9c6e09874eb685195c6d29bbc95bf3810c07738117ddfbf16c472d756 WatchSource:0}: Error finding container 1afbabc9c6e09874eb685195c6d29bbc95bf3810c07738117ddfbf16c472d756: Status 404 returned error can't find the container with id 1afbabc9c6e09874eb685195c6d29bbc95bf3810c07738117ddfbf16c472d756
Oct 29 23:40:40.358566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:40.358524    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx" event={"ID":"2c8b1696-7b99-49b0-b192-3576a4843ada","Type":"ContainerStarted","Data":"1afbabc9c6e09874eb685195c6d29bbc95bf3810c07738117ddfbf16c472d756"}
Oct 29 23:40:41.365576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:41.365485    6203 generic.go:356] "Generic (PLEG): container finished" podID="2c8b1696-7b99-49b0-b192-3576a4843ada" containerID="a07a0695cfeecb8fad1798ad3bfe136f4f6ef843090b6efc586f2988b2d65b45" exitCode=0
Oct 29 23:40:41.365576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:41.365551    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx" event={"ID":"2c8b1696-7b99-49b0-b192-3576a4843ada","Type":"ContainerDied","Data":"a07a0695cfeecb8fad1798ad3bfe136f4f6ef843090b6efc586f2988b2d65b45"}
Oct 29 23:40:42.498596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.498570    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:42.634281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.634195    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-xks-ca\") pod \"2c8b1696-7b99-49b0-b192-3576a4843ada\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") "
Oct 29 23:40:42.634448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.634321    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-openshift-ca\") pod \"2c8b1696-7b99-49b0-b192-3576a4843ada\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") "
Oct 29 23:40:42.634448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.634339    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8b1696-7b99-49b0-b192-3576a4843ada-kube-api-access-s8244\" (UniqueName: \"kubernetes.io/projected/2c8b1696-7b99-49b0-b192-3576a4843ada-kube-api-access-s8244\") pod \"2c8b1696-7b99-49b0-b192-3576a4843ada\" (UID: \"2c8b1696-7b99-49b0-b192-3576a4843ada\") "
Oct 29 23:40:42.634653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.634630    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-openshift-ca" pod "2c8b1696-7b99-49b0-b192-3576a4843ada" (UID: "2c8b1696-7b99-49b0-b192-3576a4843ada"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:40:42.636351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.636314    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-xks-ca" pod "2c8b1696-7b99-49b0-b192-3576a4843ada" (UID: "2c8b1696-7b99-49b0-b192-3576a4843ada"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:40:42.636465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.636370    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8b1696-7b99-49b0-b192-3576a4843ada-kube-api-access-s8244" pod "2c8b1696-7b99-49b0-b192-3576a4843ada" (UID: "2c8b1696-7b99-49b0-b192-3576a4843ada"). InnerVolumeSpecName "kube-api-access-s8244". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:40:42.736068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.736030    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:42.736068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.736061    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8244\" (UniqueName: \"kubernetes.io/projected/2c8b1696-7b99-49b0-b192-3576a4843ada-kube-api-access-s8244\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:42.736068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:42.736070    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c8b1696-7b99-49b0-b192-3576a4843ada-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:40:43.378387 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:43.378354    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"
Oct 29 23:40:43.378569 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:43.378388    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx" event={"ID":"2c8b1696-7b99-49b0-b192-3576a4843ada","Type":"ContainerDied","Data":"1afbabc9c6e09874eb685195c6d29bbc95bf3810c07738117ddfbf16c472d756"}
Oct 29 23:40:43.378569 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:43.378443    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1afbabc9c6e09874eb685195c6d29bbc95bf3810c07738117ddfbf16c472d756"
Oct 29 23:40:47.274782 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:47.274747    6203 scope.go:122] "RemoveContainer" containerID="c566eeaeea7a7a3ce1bbdfe0e21238af831a6c16d5bb93e6b6cfac4e7e5376df"
Oct 29 23:40:47.283918 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:47.283893    6203 scope.go:122] "RemoveContainer" containerID="b65b9e6a244f81871f48dee017cc75423e5b389c6fddf9d34e1bc382aa750d89"
Oct 29 23:40:47.292085 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:47.292064    6203 scope.go:122] "RemoveContainer" containerID="62355e36bb40fdbd61c93f5f88252300696aeaf2d6e577759b6a28164cc14998"
Oct 29 23:40:47.300640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:47.300608    6203 scope.go:122] "RemoveContainer" containerID="f7f2c5e94fcaa07b7f18c4180e3a81d6633441efa6bffec80e5070a44011c40c"
Oct 29 23:40:47.308659 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:47.308641    6203 scope.go:122] "RemoveContainer" containerID="bc85759c212353b3d9aaca72c41cf58f95b7cbaab2aa4d703b23ffde953c71c0"
Oct 29 23:40:47.316386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:47.316368    6203 scope.go:122] "RemoveContainer" containerID="45eef163b09bd1221ec5c3e6a55c120f20e747bd0718861a1e9a4050ae1641db"
Oct 29 23:40:47.324024 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:40:47.324004    6203 scope.go:122] "RemoveContainer" containerID="0247f87c2169e4fa9f49e6d0c0322b2710a808a93067981fbcef502358aa4f7f"
Oct 29 23:41:01.337667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.337570    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"]
Oct 29 23:41:01.338345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.338030    6203 cpu_manager.go:414] "containerMap: removing container" podUID="2c8b1696-7b99-49b0-b192-3576a4843ada" containerName="revoke-keys"
Oct 29 23:41:01.338345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.338045    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8b1696-7b99-49b0-b192-3576a4843ada" containerName="revoke-keys"
Oct 29 23:41:01.338345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.338141    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8b1696-7b99-49b0-b192-3576a4843ada" containerName="revoke-keys"
Oct 29 23:41:01.344030 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.344005    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.362486 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.362453    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d06645-openshift-default-dockercfg-zntvq\""
Oct 29 23:41:01.379097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.379067    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"]
Oct 29 23:41:01.416321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416282    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpjw\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-kube-api-access-5jpjw\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416322    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-credential-socket\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416344    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2788a29-47c2-42e8-8672-8a789bf705ad-istiod-ca-cert\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416469    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-envoy\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416512    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-certs\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416588    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-token\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416768 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416630    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-data\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416768 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416680    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-podinfo\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.416768 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.416715    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-socket\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517487    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-certs\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517541    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-token\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517560    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-data\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517598    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-podinfo\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517642    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-socket\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517702    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpjw\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-kube-api-access-5jpjw\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517733    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-credential-socket\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517761    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2788a29-47c2-42e8-8672-8a789bf705ad-istiod-ca-cert\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.517813 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.517803    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-envoy\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.518257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.518080    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-data\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.518257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.518150    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-credential-socket\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.518257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.518178    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-socket\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.518257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.518187    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-certs\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.518585 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.518556    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2788a29-47c2-42e8-8672-8a789bf705ad-istiod-ca-cert\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.520056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.520032    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-envoy\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.520180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.520161    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-podinfo\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.544161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.544120    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpjw\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-kube-api-access-5jpjw\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.544360 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.544266    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-token\") pod \"e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") " pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.657241 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.657144    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:01.975223 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.975197    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"]
Oct 29 23:41:01.976620 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:41:01.976586    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2788a29_47c2_42e8_8672_8a789bf705ad.slice/crio-137dea08a4f05dcaeaa0644972e5c20eeeb0729693a9feecec0d670ef044f768 WatchSource:0}: Error finding container 137dea08a4f05dcaeaa0644972e5c20eeeb0729693a9feecec0d670ef044f768: Status 404 returned error can't find the container with id 137dea08a4f05dcaeaa0644972e5c20eeeb0729693a9feecec0d670ef044f768
Oct 29 23:41:01.978906 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.978855    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:41:01.978980 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.978941    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:41:01.978980 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:01.978974    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:41:02.482679 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:02.482638    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" event={"ID":"e2788a29-47c2-42e8-8672-8a789bf705ad","Type":"ContainerStarted","Data":"2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6"}
Oct 29 23:41:02.483169 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:02.482693    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" event={"ID":"e2788a29-47c2-42e8-8672-8a789bf705ad","Type":"ContainerStarted","Data":"137dea08a4f05dcaeaa0644972e5c20eeeb0729693a9feecec0d670ef044f768"}
Oct 29 23:41:02.549334 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:02.549282    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podStartSLOduration=1.549267633 podStartE2EDuration="1.549267633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:02.547268043 +0000 UTC m=+1640.651528428" watchObservedRunningTime="2026-10-29 23:41:02.549267633 +0000 UTC m=+1640.653528020"
Oct 29 23:41:02.658152 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:02.658114    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:03.662943 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:03.662905    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:04.494049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:04.494022    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:04.495110 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:04.495087    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:06.861912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.861867    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"]
Oct 29 23:41:06.865203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.865182    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:06.885618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.885586    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"]
Oct 29 23:41:06.888119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.888096    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d06645\""
Oct 29 23:41:06.897968 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.897940    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-efcf339e-dockercfg-66xcs\""
Oct 29 23:41:06.918093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.918049    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"]
Oct 29 23:41:06.927920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.927868    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:06.949654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.949606    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"]
Oct 29 23:41:06.968844 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.968806    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaa5fa-65c7-407d-9662-e59caabdf799-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d06645-56545src6t\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:06.968844 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.968848    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbn42\" (UniqueName: \"kubernetes.io/projected/50eaa5fa-65c7-407d-9662-e59caabdf799-kube-api-access-gbn42\") pod \"payload-pre-processing-e2e-worker-single-main-d06645-56545src6t\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:06.969144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.968869    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvc8\" (UniqueName: \"kubernetes.io/projected/9f7dd383-14cd-4cb6-9942-7b59bc74413b-kube-api-access-qrvc8\") pod \"payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:06.969144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:06.969010    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd383-14cd-4cb6-9942-7b59bc74413b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:07.070415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.070381    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaa5fa-65c7-407d-9662-e59caabdf799-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d06645-56545src6t\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:07.070598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.070422    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbn42\" (UniqueName: \"kubernetes.io/projected/50eaa5fa-65c7-407d-9662-e59caabdf799-kube-api-access-gbn42\") pod \"payload-pre-processing-e2e-worker-single-main-d06645-56545src6t\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:07.070598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.070442    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvc8\" (UniqueName: \"kubernetes.io/projected/9f7dd383-14cd-4cb6-9942-7b59bc74413b-kube-api-access-qrvc8\") pod \"payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:07.070598 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.070462    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd383-14cd-4cb6-9942-7b59bc74413b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:07.071210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.071188    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd383-14cd-4cb6-9942-7b59bc74413b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:07.071252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.071206    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaa5fa-65c7-407d-9662-e59caabdf799-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d06645-56545src6t\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:07.096284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.096256    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbn42\" (UniqueName: \"kubernetes.io/projected/50eaa5fa-65c7-407d-9662-e59caabdf799-kube-api-access-gbn42\") pod \"payload-pre-processing-e2e-worker-single-main-d06645-56545src6t\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:07.096284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.096273    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvc8\" (UniqueName: \"kubernetes.io/projected/9f7dd383-14cd-4cb6-9942-7b59bc74413b-kube-api-access-qrvc8\") pod \"payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:07.179915 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.179868    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:07.245021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.244138    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:07.490193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.490169    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"]
Oct 29 23:41:07.492271 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:41:07.492243    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eaa5fa_65c7_407d_9662_e59caabdf799.slice/crio-7b73a728c6bb99314791dda5c669a8b8742df72805ab4011000ba9d4176ead8e WatchSource:0}: Error finding container 7b73a728c6bb99314791dda5c669a8b8742df72805ab4011000ba9d4176ead8e: Status 404 returned error can't find the container with id 7b73a728c6bb99314791dda5c669a8b8742df72805ab4011000ba9d4176ead8e
Oct 29 23:41:07.510258 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.510227    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" event={"ID":"50eaa5fa-65c7-407d-9662-e59caabdf799","Type":"ContainerStarted","Data":"7b73a728c6bb99314791dda5c669a8b8742df72805ab4011000ba9d4176ead8e"}
Oct 29 23:41:07.556277 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:07.556251    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"]
Oct 29 23:41:07.556619 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:41:07.556597    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7dd383_14cd_4cb6_9942_7b59bc74413b.slice/crio-3d513d5c7a3a3e434ac145d14ccb9a69c41553f6f7946e154fa5dad29a171c62 WatchSource:0}: Error finding container 3d513d5c7a3a3e434ac145d14ccb9a69c41553f6f7946e154fa5dad29a171c62: Status 404 returned error can't find the container with id 3d513d5c7a3a3e434ac145d14ccb9a69c41553f6f7946e154fa5dad29a171c62
Oct 29 23:41:08.517399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:08.517176    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" event={"ID":"9f7dd383-14cd-4cb6-9942-7b59bc74413b","Type":"ContainerStarted","Data":"72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243"}
Oct 29 23:41:08.517399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:08.517240    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" event={"ID":"9f7dd383-14cd-4cb6-9942-7b59bc74413b","Type":"ContainerStarted","Data":"3d513d5c7a3a3e434ac145d14ccb9a69c41553f6f7946e154fa5dad29a171c62"}
Oct 29 23:41:08.517399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:08.517384    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:08.518810 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:08.518784    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" event={"ID":"50eaa5fa-65c7-407d-9662-e59caabdf799","Type":"ContainerStarted","Data":"99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7"}
Oct 29 23:41:08.518911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:08.518835    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:08.564173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:08.564112    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" podStartSLOduration=2.564098171 podStartE2EDuration="2.564098171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:08.562630366 +0000 UTC m=+1646.666890753" watchObservedRunningTime="2026-10-29 23:41:08.564098171 +0000 UTC m=+1646.668358556"
Oct 29 23:41:08.608511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:08.608458    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" podStartSLOduration=2.608446716 podStartE2EDuration="2.608446716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:08.607556393 +0000 UTC m=+1646.711816780" watchObservedRunningTime="2026-10-29 23:41:08.608446716 +0000 UTC m=+1646.712707102"
Oct 29 23:41:09.925612 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:41:09.925568    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:41:19.030337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.030293    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"]
Oct 29 23:41:19.046015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.045970    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-cl2n5"]
Oct 29 23:41:19.495443 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.495356    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 29 23:41:19.495443 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.495417    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 29 23:41:19.528983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.528954    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:41:19.529129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.529026    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:41:19.609866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.609773    6203 generic.go:356] "Generic (PLEG): container finished" podID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerID="2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6" exitCode=0
Oct 29 23:41:19.609866 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.609845    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" event={"ID":"e2788a29-47c2-42e8-8672-8a789bf705ad","Type":"ContainerDied","Data":"2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6"}
Oct 29 23:41:19.610279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.610263    6203 scope.go:122] "RemoveContainer" containerID="2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6"
Oct 29 23:41:19.611524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.611490    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:41:19.611619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.611571    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:41:19.611619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:19.611611    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:41:20.021687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:20.021647    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daca662d-991e-4dcf-8c31-fc221e324171" path="/var/lib/kubelet/pods/daca662d-991e-4dcf-8c31-fc221e324171/volumes"
Oct 29 23:41:20.619009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:20.618979    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" event={"ID":"e2788a29-47c2-42e8-8672-8a789bf705ad","Type":"ContainerStarted","Data":"797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb"}
Oct 29 23:41:20.658033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:20.657997    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:20.658283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:20.658226    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 29 23:41:20.658283 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:20.658272    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 29 23:41:21.657566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:21.657527    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 29 23:41:21.657996 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:21.657582    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 29 23:41:22.469672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.469620    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"]
Oct 29 23:41:22.473136 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.473110    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.507398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.507357    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d06645\"/\"kube-root-ca.crt\""
Oct 29 23:41:22.507398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.507379    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d06645\"/\"openshift-service-ca.crt\""
Oct 29 23:41:22.509342 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.509309    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"]
Oct 29 23:41:22.519613 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.519581    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d06645\"/\"simulator-single-main-d06645-kserve-self-signed-certs\""
Oct 29 23:41:22.519783 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.519652    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d06645\"/\"default-dockercfg-bjjxj\""
Oct 29 23:41:22.578655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.578607    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-home\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.578847 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.578824    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-dshm\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.578962 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.578904    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tmp-dir\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.579050 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.579022    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tls-certs\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.579109 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.579072    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkcxn\" (UniqueName: \"kubernetes.io/projected/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-kube-api-access-nkcxn\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.579109 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.579103    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-model-cache\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.658088 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.658049    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 29 23:41:22.658471 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.658117    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 29 23:41:22.680601 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.680565    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-home\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.680773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.680609    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-dshm\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.680773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.680745    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tmp-dir\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.680873 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.680850    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tls-certs\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.680948 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.680901    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkcxn\" (UniqueName: \"kubernetes.io/projected/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-kube-api-access-nkcxn\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.680948 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.680929    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-model-cache\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.681060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.681033    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-home\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.681139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.681112    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tmp-dir\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.681269 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.681250    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-model-cache\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.683109 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.683086    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-dshm\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.683442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.683425    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tls-certs\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.711543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.711504    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkcxn\" (UniqueName: \"kubernetes.io/projected/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-kube-api-access-nkcxn\") pod \"simulator-single-main-d06645-kserve-69949b889c-7zkjl\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") " pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:22.795146 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:22.794936    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:23.126904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:23.126867    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"]
Oct 29 23:41:23.658056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:23.658013    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 29 23:41:23.658238 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:23.658091    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 29 23:41:23.659592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:23.659562    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl" event={"ID":"a8438f10-a579-4dfb-b40a-27dd8a4eaa48","Type":"ContainerStarted","Data":"a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b"}
Oct 29 23:41:23.659693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:23.659606    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl" event={"ID":"a8438f10-a579-4dfb-b40a-27dd8a4eaa48","Type":"ContainerStarted","Data":"1aa09e78aacf6e6549899ca2db4820af04f60c9a733f663ea120873da0c2303f"}
Oct 29 23:41:23.754891 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:23.754810    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl" podStartSLOduration=1.75478936 podStartE2EDuration="1.75478936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:23.753385507 +0000 UTC m=+1661.857645894" watchObservedRunningTime="2026-10-29 23:41:23.75478936 +0000 UTC m=+1661.859049749"
Oct 29 23:41:23.795559 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:23.795472    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:24.658521 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:24.658480    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 29 23:41:24.658978 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:24.658536    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 29 23:41:25.657622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:25.657581    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 29 23:41:25.657817 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:25.657655    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 29 23:41:26.661800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:26.661764    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:26.662230 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:26.662207    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:26.663045 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:26.663024    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:41:32.796065 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:32.796025    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:32.809242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:32.809212    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:33.727729 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:33.727668    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:41:39.928425 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:41:39.928391    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:41:45.620942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.620910    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"]
Oct 29 23:41:45.624164 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.624134    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.641677 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.641655    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:41:45.656464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.656430    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:41:45.656595 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.656455    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:41:45.656595 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.656455    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:41:45.658741 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.658720    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"]
Oct 29 23:41:45.718267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.718237    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x85g\" (UniqueName: \"kubernetes.io/projected/d0792e31-57fc-4172-9956-e8c1a289f739-kube-api-access-4x85g\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.718445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.718366    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.718445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.718427    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.819655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.819618    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.819655 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.819663    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.819949 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.819725    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x85g\" (UniqueName: \"kubernetes.io/projected/d0792e31-57fc-4172-9956-e8c1a289f739-kube-api-access-4x85g\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.820350 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.820323    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.822142 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.822119    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.846137 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.846108    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x85g\" (UniqueName: \"kubernetes.io/projected/d0792e31-57fc-4172-9956-e8c1a289f739-kube-api-access-4x85g\") pod \"maas-api-revoke-sub-33257fa5-knntm\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:45.936200 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:45.936173    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:46.242787 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:46.242759    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"]
Oct 29 23:41:46.243855 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:41:46.243830    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0792e31_57fc_4172_9956_e8c1a289f739.slice/crio-cc4b6108915a86658e2c44bb68b9c8755f16d3296010b325301e9d953e9f35ed WatchSource:0}: Error finding container cc4b6108915a86658e2c44bb68b9c8755f16d3296010b325301e9d953e9f35ed: Status 404 returned error can't find the container with id cc4b6108915a86658e2c44bb68b9c8755f16d3296010b325301e9d953e9f35ed
Oct 29 23:41:46.791307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:46.791222    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm" event={"ID":"d0792e31-57fc-4172-9956-e8c1a289f739","Type":"ContainerStarted","Data":"bce47be3e69709af02dbb50c84bc1bcb7bff3b7d73894d3f8f6f1f3d9b6f0440"}
Oct 29 23:41:46.791307 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:46.791272    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm" event={"ID":"d0792e31-57fc-4172-9956-e8c1a289f739","Type":"ContainerStarted","Data":"cc4b6108915a86658e2c44bb68b9c8755f16d3296010b325301e9d953e9f35ed"}
Oct 29 23:41:46.843018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:46.842967    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm" podStartSLOduration=1.842949191 podStartE2EDuration="1.842949191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:46.842100514 +0000 UTC m=+1684.946360900" watchObservedRunningTime="2026-10-29 23:41:46.842949191 +0000 UTC m=+1684.947209574"
Oct 29 23:41:47.378621 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.378584    6203 scope.go:122] "RemoveContainer" containerID="1bd670ba5a328e88abf23bd5cc1050deb9ff7390b70860815638c344edf42bc7"
Oct 29 23:41:47.388002 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.387975    6203 scope.go:122] "RemoveContainer" containerID="68e5f889bcfb6d4ca86c9d3658e86d0c1f1515bb0cda315e53826f76cdd178a2"
Oct 29 23:41:47.395920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.395897    6203 scope.go:122] "RemoveContainer" containerID="7b750667a8864dcca25fd7e2fa27663b2d8d441ad6bbbb8fbfcda33d10ce2049"
Oct 29 23:41:47.403843 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.403821    6203 scope.go:122] "RemoveContainer" containerID="fc032eee016705e5a9bde975f243d79a3c7f1931ae98641e86efc47e6017f3bd"
Oct 29 23:41:47.411622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.411605    6203 scope.go:122] "RemoveContainer" containerID="11876eaf8c49f065a9b27cf87a7ff4d798e86f1b23d461ce66d0f15efeeb73df"
Oct 29 23:41:47.419319 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.419300    6203 scope.go:122] "RemoveContainer" containerID="e459e4c0b21350ad2d9f4d32dc0ed8bdb571fbc079a08efb9244bd9fa113c809"
Oct 29 23:41:47.426757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.426740    6203 scope.go:122] "RemoveContainer" containerID="6c7cd96b65dfa76ac9153147f9448e03c3599a57d2a24b1d0cbff82ca61aafd9"
Oct 29 23:41:47.434396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.434380    6203 scope.go:122] "RemoveContainer" containerID="535f77e20901a705235d31870d308a406f7aed56b2c5611f307a99ee543f6003"
Oct 29 23:41:47.798480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.798441    6203 generic.go:356] "Generic (PLEG): container finished" podID="d0792e31-57fc-4172-9956-e8c1a289f739" containerID="bce47be3e69709af02dbb50c84bc1bcb7bff3b7d73894d3f8f6f1f3d9b6f0440" exitCode=0
Oct 29 23:41:47.798480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:47.798482    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm" event={"ID":"d0792e31-57fc-4172-9956-e8c1a289f739","Type":"ContainerDied","Data":"bce47be3e69709af02dbb50c84bc1bcb7bff3b7d73894d3f8f6f1f3d9b6f0440"}
Oct 29 23:41:48.938152 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:48.938122    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:49.053124 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.053087    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-xks-ca\") pod \"d0792e31-57fc-4172-9956-e8c1a289f739\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") "
Oct 29 23:41:49.053315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.053174    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-openshift-ca\") pod \"d0792e31-57fc-4172-9956-e8c1a289f739\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") "
Oct 29 23:41:49.053315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.053200    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0792e31-57fc-4172-9956-e8c1a289f739-kube-api-access-4x85g\" (UniqueName: \"kubernetes.io/projected/d0792e31-57fc-4172-9956-e8c1a289f739-kube-api-access-4x85g\") pod \"d0792e31-57fc-4172-9956-e8c1a289f739\" (UID: \"d0792e31-57fc-4172-9956-e8c1a289f739\") "
Oct 29 23:41:49.053613 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.053586    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-openshift-ca" pod "d0792e31-57fc-4172-9956-e8c1a289f739" (UID: "d0792e31-57fc-4172-9956-e8c1a289f739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:41:49.055347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.055292    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-xks-ca" pod "d0792e31-57fc-4172-9956-e8c1a289f739" (UID: "d0792e31-57fc-4172-9956-e8c1a289f739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:41:49.055347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.055327    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0792e31-57fc-4172-9956-e8c1a289f739-kube-api-access-4x85g" pod "d0792e31-57fc-4172-9956-e8c1a289f739" (UID: "d0792e31-57fc-4172-9956-e8c1a289f739"). InnerVolumeSpecName "kube-api-access-4x85g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:41:49.154462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.154430    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:41:49.154462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.154458    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0792e31-57fc-4172-9956-e8c1a289f739-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:41:49.154462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.154468    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x85g\" (UniqueName: \"kubernetes.io/projected/d0792e31-57fc-4172-9956-e8c1a289f739-kube-api-access-4x85g\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:41:49.810611 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.810578    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm" event={"ID":"d0792e31-57fc-4172-9956-e8c1a289f739","Type":"ContainerDied","Data":"cc4b6108915a86658e2c44bb68b9c8755f16d3296010b325301e9d953e9f35ed"}
Oct 29 23:41:49.810794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.810616    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"
Oct 29 23:41:49.810794 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:49.810629    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc4b6108915a86658e2c44bb68b9c8755f16d3296010b325301e9d953e9f35ed"
Oct 29 23:41:56.804494 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.804450    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"]
Oct 29 23:41:56.805126 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.805105    6203 cpu_manager.go:414] "containerMap: removing container" podUID="d0792e31-57fc-4172-9956-e8c1a289f739" containerName="revoke-keys"
Oct 29 23:41:56.805227 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.805128    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0792e31-57fc-4172-9956-e8c1a289f739" containerName="revoke-keys"
Oct 29 23:41:56.805280 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.805242    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0792e31-57fc-4172-9956-e8c1a289f739" containerName="revoke-keys"
Oct 29 23:41:56.808533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.808509    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:56.829175 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.829143    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"]
Oct 29 23:41:56.840590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.840562    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:41:56.840755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.840563    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:41:56.840755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.840640    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:41:56.840755 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.840692    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:41:56.929589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.929558    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xr64\" (UniqueName: \"kubernetes.io/projected/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-kube-api-access-4xr64\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:56.929589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.929593    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:56.929808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:56.929614    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.030963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.030920    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xr64\" (UniqueName: \"kubernetes.io/projected/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-kube-api-access-4xr64\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.031147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.030976    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.031147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.031008    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.031786 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.031757    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.033391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.033367    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.056809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.056746    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xr64\" (UniqueName: \"kubernetes.io/projected/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-kube-api-access-4xr64\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.120957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.120923    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:41:57.432670 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.432640    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"]
Oct 29 23:41:57.434412 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:41:57.434380    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda743ac_4e6e_47d7_a0db_71e3ab8e6d3f.slice/crio-9fe7aaea46ef48b344ed510199ebb0a1b92f8b7bb37d8270c5789279dc1c5061 WatchSource:0}: Error finding container 9fe7aaea46ef48b344ed510199ebb0a1b92f8b7bb37d8270c5789279dc1c5061: Status 404 returned error can't find the container with id 9fe7aaea46ef48b344ed510199ebb0a1b92f8b7bb37d8270c5789279dc1c5061
Oct 29 23:41:57.855816 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.855775    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x" event={"ID":"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f","Type":"ContainerStarted","Data":"be80b127ec230c20ba3a360eb25005dfe9c039ddace9f785e29418d29cfc4902"}
Oct 29 23:41:57.856236 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.855838    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x" event={"ID":"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f","Type":"ContainerStarted","Data":"9fe7aaea46ef48b344ed510199ebb0a1b92f8b7bb37d8270c5789279dc1c5061"}
Oct 29 23:41:57.909272 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:57.909217    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x" podStartSLOduration=1.909203317 podStartE2EDuration="1.909203317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:41:57.907437076 +0000 UTC m=+1696.011697463" watchObservedRunningTime="2026-10-29 23:41:57.909203317 +0000 UTC m=+1696.013463703"
Oct 29 23:41:58.863243 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:58.863202    6203 generic.go:356] "Generic (PLEG): container finished" podID="dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" containerID="be80b127ec230c20ba3a360eb25005dfe9c039ddace9f785e29418d29cfc4902" exitCode=0
Oct 29 23:41:58.863686 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:41:58.863301    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x" event={"ID":"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f","Type":"ContainerDied","Data":"be80b127ec230c20ba3a360eb25005dfe9c039ddace9f785e29418d29cfc4902"}
Oct 29 23:42:00.008097 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.008063    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:42:00.165249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.165161    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-xks-ca\") pod \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") "
Oct 29 23:42:00.165249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.165219    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-kube-api-access-4xr64\" (UniqueName: \"kubernetes.io/projected/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-kube-api-access-4xr64\") pod \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") "
Oct 29 23:42:00.165467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.165419    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-openshift-ca\") pod \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\" (UID: \"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f\") "
Oct 29 23:42:00.165851 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.165820    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-openshift-ca" pod "dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" (UID: "dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:00.167335 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.167313    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-xks-ca" pod "dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" (UID: "dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:42:00.167506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.167483    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-kube-api-access-4xr64" pod "dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" (UID: "dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f"). InnerVolumeSpecName "kube-api-access-4xr64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:00.266401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.266352    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:00.266401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.266392    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:00.266401 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.266402    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xr64\" (UniqueName: \"kubernetes.io/projected/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f-kube-api-access-4xr64\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:00.877308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.877276    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x" event={"ID":"dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f","Type":"ContainerDied","Data":"9fe7aaea46ef48b344ed510199ebb0a1b92f8b7bb37d8270c5789279dc1c5061"}
Oct 29 23:42:00.877520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.877333    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fe7aaea46ef48b344ed510199ebb0a1b92f8b7bb37d8270c5789279dc1c5061"
Oct 29 23:42:00.877520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:00.877283    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"
Oct 29 23:42:01.600329 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:01.600294    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"]
Oct 29 23:42:07.291725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:07.291685    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"]
Oct 29 23:42:07.325350 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:07.325314    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"]
Oct 29 23:42:08.028752 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:08.028720    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"]
Oct 29 23:42:08.031262 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:08.031240    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ghn86"]
Oct 29 23:42:09.783701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:09.783660    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"]
Oct 29 23:42:09.801019 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:09.800981    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d06645-9z24x"]
Oct 29 23:42:09.923742 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:42:09.923700    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:42:10.017468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:10.017424    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed3c848-038c-453a-8b66-6fb68d2d8dc4" path="/var/lib/kubelet/pods/1ed3c848-038c-453a-8b66-6fb68d2d8dc4/volumes"
Oct 29 23:42:10.018000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:10.017978    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" path="/var/lib/kubelet/pods/dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f/volumes"
Oct 29 23:42:12.292863 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:12.292797    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" podUID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" containerName="payload-processing" containerID="cri-o://72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243" gracePeriod=25
Oct 29 23:42:12.326540 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:12.326498    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" podUID="50eaa5fa-65c7-407d-9662-e59caabdf799" containerName="payload-pre-processing" containerID="cri-o://99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7" gracePeriod=25
Oct 29 23:42:16.568355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.568321    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"]
Oct 29 23:42:16.568852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.568585    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy" containerID="cri-o://797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb" gracePeriod=30
Oct 29 23:42:16.620341 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.620303    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl" podUID="a8438f10-a579-4dfb-b40a-27dd8a4eaa48" containerName="main" containerID="cri-o://a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b" gracePeriod=45
Oct 29 23:42:16.716892 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.716851    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl" podUID="a8438f10-a579-4dfb-b40a-27dd8a4eaa48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.155:8000/ready\": dial tcp 10.134.0.155:8000: connect: connection refused"
Oct 29 23:42:16.881695 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.881661    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:42:16.914470 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914437    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tmp-dir\") pod \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") "
Oct 29 23:42:16.914656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914488    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-home\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-home\") pod \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") "
Oct 29 23:42:16.914656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914522    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-dshm\") pod \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") "
Oct 29 23:42:16.914656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914562    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tls-certs\") pod \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") "
Oct 29 23:42:16.914656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914609    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-kube-api-access-nkcxn\" (UniqueName: \"kubernetes.io/projected/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-kube-api-access-nkcxn\") pod \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") "
Oct 29 23:42:16.914896 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914699    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-model-cache\") pod \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\" (UID: \"a8438f10-a579-4dfb-b40a-27dd8a4eaa48\") "
Oct 29 23:42:16.914896 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914755    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tmp-dir" pod "a8438f10-a579-4dfb-b40a-27dd8a4eaa48" (UID: "a8438f10-a579-4dfb-b40a-27dd8a4eaa48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:16.914896 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.914805    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-home" pod "a8438f10-a579-4dfb-b40a-27dd8a4eaa48" (UID: "a8438f10-a579-4dfb-b40a-27dd8a4eaa48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:16.915077 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.915032    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-model-cache" pod "a8438f10-a579-4dfb-b40a-27dd8a4eaa48" (UID: "a8438f10-a579-4dfb-b40a-27dd8a4eaa48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:16.915121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.915087    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:16.915121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.915108    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:16.916904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.916866    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-dshm" pod "a8438f10-a579-4dfb-b40a-27dd8a4eaa48" (UID: "a8438f10-a579-4dfb-b40a-27dd8a4eaa48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:16.917014 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.916919    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tls-certs" pod "a8438f10-a579-4dfb-b40a-27dd8a4eaa48" (UID: "a8438f10-a579-4dfb-b40a-27dd8a4eaa48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:42:16.917072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.917026    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-kube-api-access-nkcxn" pod "a8438f10-a579-4dfb-b40a-27dd8a4eaa48" (UID: "a8438f10-a579-4dfb-b40a-27dd8a4eaa48"). InnerVolumeSpecName "kube-api-access-nkcxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:16.971512 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.971468    6203 generic.go:356] "Generic (PLEG): container finished" podID="a8438f10-a579-4dfb-b40a-27dd8a4eaa48" containerID="a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b" exitCode=0
Oct 29 23:42:16.971697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.971514    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl" event={"ID":"a8438f10-a579-4dfb-b40a-27dd8a4eaa48","Type":"ContainerDied","Data":"a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b"}
Oct 29 23:42:16.971697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.971536    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"
Oct 29 23:42:16.971697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.971561    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl" event={"ID":"a8438f10-a579-4dfb-b40a-27dd8a4eaa48","Type":"ContainerDied","Data":"1aa09e78aacf6e6549899ca2db4820af04f60c9a733f663ea120873da0c2303f"}
Oct 29 23:42:16.971697 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.971590    6203 scope.go:122] "RemoveContainer" containerID="a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b"
Oct 29 23:42:16.981232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.981209    6203 scope.go:122] "RemoveContainer" containerID="a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b"
Oct 29 23:42:16.981532 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:42:16.981510    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b\": container with ID starting with a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b not found: ID does not exist" containerID="a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b"
Oct 29 23:42:16.981603 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:16.981542    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b"} err="failed to get container status \"a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b\": rpc error: code = NotFound desc = could not find container \"a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b\": container with ID starting with a36e64fada690e441ed06ceb23285fa17d405e6e755f9a81ece961e1400cbd8b not found: ID does not exist"
Oct 29 23:42:17.015762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.015720    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:17.015762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.015753    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:17.015762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.015766    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:17.016036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.015777    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkcxn\" (UniqueName: \"kubernetes.io/projected/a8438f10-a579-4dfb-b40a-27dd8a4eaa48-kube-api-access-nkcxn\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:17.030690 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.030659    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"]
Oct 29 23:42:17.039679 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.039649    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d06645/simulator-single-main-d06645-kserve-69949b889c-7zkjl"]
Oct 29 23:42:17.794975 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.794942    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"]
Oct 29 23:42:17.809627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.809584    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tszfx"]
Oct 29 23:42:17.845649 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.845616    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"]
Oct 29 23:42:17.846237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.846218    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a8438f10-a579-4dfb-b40a-27dd8a4eaa48" containerName="main"
Oct 29 23:42:17.846288 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.846240    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8438f10-a579-4dfb-b40a-27dd8a4eaa48" containerName="main"
Oct 29 23:42:17.846288 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.846267    6203 cpu_manager.go:414] "containerMap: removing container" podUID="dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" containerName="revoke-keys"
Oct 29 23:42:17.846288 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.846274    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" containerName="revoke-keys"
Oct 29 23:42:17.846407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.846395    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda743ac-4e6e-47d7-a0db-71e3ab8e6d3f" containerName="revoke-keys"
Oct 29 23:42:17.846446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.846419    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8438f10-a579-4dfb-b40a-27dd8a4eaa48" containerName="main"
Oct 29 23:42:17.851121 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.851101    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:17.874577 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.874525    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:42:17.874834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.874617    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:42:17.874834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.874768    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:42:17.875118 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.875094    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:42:17.881237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.881200    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"]
Oct 29 23:42:17.925556 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.925524    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds77r\" (UniqueName: \"kubernetes.io/projected/746c044c-105d-4d84-bb5d-e16ba4c3cdce-kube-api-access-ds77r\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:17.925724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.925633    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:17.925724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:17.925702    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.017271 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.017240    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8b1696-7b99-49b0-b192-3576a4843ada" path="/var/lib/kubelet/pods/2c8b1696-7b99-49b0-b192-3576a4843ada/volumes"
Oct 29 23:42:18.017666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.017653    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8438f10-a579-4dfb-b40a-27dd8a4eaa48" path="/var/lib/kubelet/pods/a8438f10-a579-4dfb-b40a-27dd8a4eaa48/volumes"
Oct 29 23:42:18.026397 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.026371    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds77r\" (UniqueName: \"kubernetes.io/projected/746c044c-105d-4d84-bb5d-e16ba4c3cdce-kube-api-access-ds77r\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.026554 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.026528    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.026618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.026581    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.027354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.027325    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.029035 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.029012    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.064854 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.064773    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds77r\" (UniqueName: \"kubernetes.io/projected/746c044c-105d-4d84-bb5d-e16ba4c3cdce-kube-api-access-ds77r\") pod \"maas-api-revoke-sub-3d4156f1-gk5pf\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.164807 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.164757    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:18.468156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.468129    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"]
Oct 29 23:42:18.470244 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:42:18.470211    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746c044c_105d_4d84_bb5d_e16ba4c3cdce.slice/crio-4f5bb30232609bed06292f37a754e02b0025a298e0752f4db6be670fb938f4fa WatchSource:0}: Error finding container 4f5bb30232609bed06292f37a754e02b0025a298e0752f4db6be670fb938f4fa: Status 404 returned error can't find the container with id 4f5bb30232609bed06292f37a754e02b0025a298e0752f4db6be670fb938f4fa
Oct 29 23:42:18.985138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.985102    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf" event={"ID":"746c044c-105d-4d84-bb5d-e16ba4c3cdce","Type":"ContainerStarted","Data":"b0470cdd87a27fb606ee243b09f8d679cd9c9074d34eecbedb8bad093bf21205"}
Oct 29 23:42:18.985535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:18.985158    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf" event={"ID":"746c044c-105d-4d84-bb5d-e16ba4c3cdce","Type":"ContainerStarted","Data":"4f5bb30232609bed06292f37a754e02b0025a298e0752f4db6be670fb938f4fa"}
Oct 29 23:42:19.030535 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:19.030450    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf" podStartSLOduration=2.030437529 podStartE2EDuration="2.030437529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:42:19.030012985 +0000 UTC m=+1717.134273370" watchObservedRunningTime="2026-10-29 23:42:19.030437529 +0000 UTC m=+1717.134697915"
Oct 29 23:42:19.527589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:19.527548    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" podUID="50eaa5fa-65c7-407d-9662-e59caabdf799" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:19.527766 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:19.527600    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" podUID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:19.998926 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:19.998873    6203 generic.go:356] "Generic (PLEG): container finished" podID="746c044c-105d-4d84-bb5d-e16ba4c3cdce" containerID="b0470cdd87a27fb606ee243b09f8d679cd9c9074d34eecbedb8bad093bf21205" exitCode=0
Oct 29 23:42:19.999305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:19.998974    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf" event={"ID":"746c044c-105d-4d84-bb5d-e16ba4c3cdce","Type":"ContainerDied","Data":"b0470cdd87a27fb606ee243b09f8d679cd9c9074d34eecbedb8bad093bf21205"}
Oct 29 23:42:21.144183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.144155    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:21.258561 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.258477    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746c044c-105d-4d84-bb5d-e16ba4c3cdce-kube-api-access-ds77r\" (UniqueName: \"kubernetes.io/projected/746c044c-105d-4d84-bb5d-e16ba4c3cdce-kube-api-access-ds77r\") pod \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") "
Oct 29 23:42:21.258725 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.258662    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-openshift-ca\") pod \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") "
Oct 29 23:42:21.258792 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.258727    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-xks-ca\") pod \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\" (UID: \"746c044c-105d-4d84-bb5d-e16ba4c3cdce\") "
Oct 29 23:42:21.259072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.259046    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-openshift-ca" pod "746c044c-105d-4d84-bb5d-e16ba4c3cdce" (UID: "746c044c-105d-4d84-bb5d-e16ba4c3cdce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:21.260693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.260663    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746c044c-105d-4d84-bb5d-e16ba4c3cdce-kube-api-access-ds77r" pod "746c044c-105d-4d84-bb5d-e16ba4c3cdce" (UID: "746c044c-105d-4d84-bb5d-e16ba4c3cdce"). InnerVolumeSpecName "kube-api-access-ds77r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:21.260693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.260662    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-xks-ca" pod "746c044c-105d-4d84-bb5d-e16ba4c3cdce" (UID: "746c044c-105d-4d84-bb5d-e16ba4c3cdce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:42:21.359654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.359620    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.359654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.359651    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds77r\" (UniqueName: \"kubernetes.io/projected/746c044c-105d-4d84-bb5d-e16ba4c3cdce-kube-api-access-ds77r\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.359654 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.359661    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746c044c-105d-4d84-bb5d-e16ba4c3cdce-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.805355 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.805328    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:42:21.864079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864002    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2788a29-47c2-42e8-8672-8a789bf705ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2788a29-47c2-42e8-8672-8a789bf705ad-istiod-ca-cert\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864038    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-socket\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864062    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-envoy\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864118    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-credential-socket\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864179    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-podinfo\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864211    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-kube-api-access-5jpjw\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-kube-api-access-5jpjw\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864239    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-token\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-token\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864335    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-data\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864397    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-certs\") pod \"e2788a29-47c2-42e8-8672-8a789bf705ad\" (UID: \"e2788a29-47c2-42e8-8672-8a789bf705ad\") "
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864402    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-socket" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864472    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-credential-socket" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:21.864557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864536    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2788a29-47c2-42e8-8672-8a789bf705ad-istiod-ca-cert" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:21.865074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864753    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-certs" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:21.865074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864819    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2788a29-47c2-42e8-8672-8a789bf705ad-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.865074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864841    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.865074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864856    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.865074 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.864901    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-data" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:21.866459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.866431    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-podinfo" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:42:21.866698 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.866665    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-kube-api-access-5jpjw" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "kube-api-access-5jpjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:21.866808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.866677    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-envoy" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:42:21.866808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.866799    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-token" pod "e2788a29-47c2-42e8-8672-8a789bf705ad" (UID: "e2788a29-47c2-42e8-8672-8a789bf705ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:21.965759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.965728    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.965759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.965753    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.965759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.965762    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jpjw\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-kube-api-access-5jpjw\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.966023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.965771    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.966023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.965779    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:21.966023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:21.965787    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2788a29-47c2-42e8-8672-8a789bf705ad-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:22.010764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.010733    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"
Oct 29 23:42:22.010956 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.010763    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf" event={"ID":"746c044c-105d-4d84-bb5d-e16ba4c3cdce","Type":"ContainerDied","Data":"4f5bb30232609bed06292f37a754e02b0025a298e0752f4db6be670fb938f4fa"}
Oct 29 23:42:22.010956 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.010815    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f5bb30232609bed06292f37a754e02b0025a298e0752f4db6be670fb938f4fa"
Oct 29 23:42:22.012718 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.012694    6203 generic.go:356] "Generic (PLEG): container finished" podID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerID="797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb" exitCode=0
Oct 29 23:42:22.015468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.015443    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"
Oct 29 23:42:22.016848 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.016825    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" event={"ID":"e2788a29-47c2-42e8-8672-8a789bf705ad","Type":"ContainerDied","Data":"797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb"}
Oct 29 23:42:22.016968 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.016915    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx" event={"ID":"e2788a29-47c2-42e8-8672-8a789bf705ad","Type":"ContainerDied","Data":"137dea08a4f05dcaeaa0644972e5c20eeeb0729693a9feecec0d670ef044f768"}
Oct 29 23:42:22.016968 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.016939    6203 scope.go:122] "RemoveContainer" containerID="797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb"
Oct 29 23:42:22.027856 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.027843    6203 scope.go:122] "RemoveContainer" containerID="2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6"
Oct 29 23:42:22.036170 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.036154    6203 scope.go:122] "RemoveContainer" containerID="797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb"
Oct 29 23:42:22.036422 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:42:22.036403    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb\": container with ID starting with 797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb not found: ID does not exist" containerID="797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb"
Oct 29 23:42:22.036466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.036430    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb"} err="failed to get container status \"797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb\": rpc error: code = NotFound desc = could not find container \"797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb\": container with ID starting with 797c32c3b5fb1f519b6a443ff3752574940ff78f835426dbefc1f02153dad2cb not found: ID does not exist"
Oct 29 23:42:22.036466 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.036445    6203 scope.go:122] "RemoveContainer" containerID="2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6"
Oct 29 23:42:22.036677 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:42:22.036655    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6\": container with ID starting with 2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6 not found: ID does not exist" containerID="2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6"
Oct 29 23:42:22.036722 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.036684    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6"} err="failed to get container status \"2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6\": rpc error: code = NotFound desc = could not find container \"2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6\": container with ID starting with 2b6b27d424bd1a1bc14f2a3f454d0eecde5174160f7bdbe8c90ac6d53397aab6 not found: ID does not exist"
Oct 29 23:42:22.090889 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.090855    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"]
Oct 29 23:42:22.097672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:22.097643    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d06645-openshift-default-79fd8b88ccx57kx"]
Oct 29 23:42:24.016967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:24.016873    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" path="/var/lib/kubelet/pods/e2788a29-47c2-42e8-8672-8a789bf705ad/volumes"
Oct 29 23:42:29.527622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:29.527585    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" podUID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:29.528042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:29.527594    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" podUID="50eaa5fa-65c7-407d-9662-e59caabdf799" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:42:32.018061 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.018021    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-rwn2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:42:32.018462 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.018108    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-rwn2k" podUID="9b2c909d-461e-4fff-9d3f-af4bc893adc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:42:32.029995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.029963    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"]
Oct 29 23:42:32.046018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.045978    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-k7vqh"]
Oct 29 23:42:32.581284 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.581253    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:42:32.606799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.606776    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:42:32.707935 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.707901    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50eaa5fa-65c7-407d-9662-e59caabdf799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaa5fa-65c7-407d-9662-e59caabdf799-plugins-config-volume\") pod \"50eaa5fa-65c7-407d-9662-e59caabdf799\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") "
Oct 29 23:42:32.708082 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.707963    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50eaa5fa-65c7-407d-9662-e59caabdf799-kube-api-access-gbn42\" (UniqueName: \"kubernetes.io/projected/50eaa5fa-65c7-407d-9662-e59caabdf799-kube-api-access-gbn42\") pod \"50eaa5fa-65c7-407d-9662-e59caabdf799\" (UID: \"50eaa5fa-65c7-407d-9662-e59caabdf799\") "
Oct 29 23:42:32.708142 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.708072    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7dd383-14cd-4cb6-9942-7b59bc74413b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd383-14cd-4cb6-9942-7b59bc74413b-plugins-config-volume\") pod \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") "
Oct 29 23:42:32.708226 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.708208    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7dd383-14cd-4cb6-9942-7b59bc74413b-kube-api-access-qrvc8\" (UniqueName: \"kubernetes.io/projected/9f7dd383-14cd-4cb6-9942-7b59bc74413b-kube-api-access-qrvc8\") pod \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\" (UID: \"9f7dd383-14cd-4cb6-9942-7b59bc74413b\") "
Oct 29 23:42:32.708332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.708312    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50eaa5fa-65c7-407d-9662-e59caabdf799-plugins-config-volume" pod "50eaa5fa-65c7-407d-9662-e59caabdf799" (UID: "50eaa5fa-65c7-407d-9662-e59caabdf799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:32.708486 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.708466    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7dd383-14cd-4cb6-9942-7b59bc74413b-plugins-config-volume" pod "9f7dd383-14cd-4cb6-9942-7b59bc74413b" (UID: "9f7dd383-14cd-4cb6-9942-7b59bc74413b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:42:32.708563 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.708513    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50eaa5fa-65c7-407d-9662-e59caabdf799-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:32.710025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.709993    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eaa5fa-65c7-407d-9662-e59caabdf799-kube-api-access-gbn42" pod "50eaa5fa-65c7-407d-9662-e59caabdf799" (UID: "50eaa5fa-65c7-407d-9662-e59caabdf799"). InnerVolumeSpecName "kube-api-access-gbn42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:32.710174 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.710153    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7dd383-14cd-4cb6-9942-7b59bc74413b-kube-api-access-qrvc8" pod "9f7dd383-14cd-4cb6-9942-7b59bc74413b" (UID: "9f7dd383-14cd-4cb6-9942-7b59bc74413b"). InnerVolumeSpecName "kube-api-access-qrvc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:42:32.810072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.809994    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbn42\" (UniqueName: \"kubernetes.io/projected/50eaa5fa-65c7-407d-9662-e59caabdf799-kube-api-access-gbn42\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:32.810072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.810020    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd383-14cd-4cb6-9942-7b59bc74413b-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:32.810072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:32.810031    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvc8\" (UniqueName: \"kubernetes.io/projected/9f7dd383-14cd-4cb6-9942-7b59bc74413b-kube-api-access-qrvc8\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:42:33.107122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.107026    6203 generic.go:356] "Generic (PLEG): container finished" podID="50eaa5fa-65c7-407d-9662-e59caabdf799" containerID="99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7" exitCode=0
Oct 29 23:42:33.107122 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.107084    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"
Oct 29 23:42:33.107606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.107116    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" event={"ID":"50eaa5fa-65c7-407d-9662-e59caabdf799","Type":"ContainerDied","Data":"99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7"}
Oct 29 23:42:33.107606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.107176    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t" event={"ID":"50eaa5fa-65c7-407d-9662-e59caabdf799","Type":"ContainerDied","Data":"7b73a728c6bb99314791dda5c669a8b8742df72805ab4011000ba9d4176ead8e"}
Oct 29 23:42:33.107606 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.107203    6203 scope.go:122] "RemoveContainer" containerID="99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7"
Oct 29 23:42:33.108776 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.108746    6203 generic.go:356] "Generic (PLEG): container finished" podID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" containerID="72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243" exitCode=0
Oct 29 23:42:33.108861 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.108781    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" event={"ID":"9f7dd383-14cd-4cb6-9942-7b59bc74413b","Type":"ContainerDied","Data":"72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243"}
Oct 29 23:42:33.108861 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.108797    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"
Oct 29 23:42:33.108861 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.108821    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths" event={"ID":"9f7dd383-14cd-4cb6-9942-7b59bc74413b","Type":"ContainerDied","Data":"3d513d5c7a3a3e434ac145d14ccb9a69c41553f6f7946e154fa5dad29a171c62"}
Oct 29 23:42:33.116988 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.116971    6203 scope.go:122] "RemoveContainer" containerID="99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7"
Oct 29 23:42:33.117251 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:42:33.117228    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7\": container with ID starting with 99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7 not found: ID does not exist" containerID="99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7"
Oct 29 23:42:33.117347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.117258    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7"} err="failed to get container status \"99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7\": rpc error: code = NotFound desc = could not find container \"99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7\": container with ID starting with 99f5c293a5f64424085a648ceec1aae6eb391d6003c33fdfb1de76c4f6595be7 not found: ID does not exist"
Oct 29 23:42:33.117347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.117276    6203 scope.go:122] "RemoveContainer" containerID="72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243"
Oct 29 23:42:33.125638 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.125622    6203 scope.go:122] "RemoveContainer" containerID="72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243"
Oct 29 23:42:33.125940 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:42:33.125922    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243\": container with ID starting with 72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243 not found: ID does not exist" containerID="72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243"
Oct 29 23:42:33.126013 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.125951    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243"} err="failed to get container status \"72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243\": rpc error: code = NotFound desc = could not find container \"72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243\": container with ID starting with 72754369c80e4e00821398fd749d24f6bb41821aa8575cefa2ec02e443753243 not found: ID does not exist"
Oct 29 23:42:33.165591 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.165555    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"]
Oct 29 23:42:33.180367 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.180336    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d06645-56545src6t"]
Oct 29 23:42:33.208726 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.208694    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"]
Oct 29 23:42:33.213455 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:33.213430    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d06645-c57bd684b5vths"]
Oct 29 23:42:34.017413 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:34.017382    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50eaa5fa-65c7-407d-9662-e59caabdf799" path="/var/lib/kubelet/pods/50eaa5fa-65c7-407d-9662-e59caabdf799/volumes"
Oct 29 23:42:34.017793 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:34.017780    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e192588-39c1-461b-92e7-26a8f05a4f17" path="/var/lib/kubelet/pods/8e192588-39c1-461b-92e7-26a8f05a4f17/volumes"
Oct 29 23:42:34.018155 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:34.018141    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" path="/var/lib/kubelet/pods/9f7dd383-14cd-4cb6-9942-7b59bc74413b/volumes"
Oct 29 23:42:39.922383 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:42:39.922347    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:42:43.028961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:43.028916    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"]
Oct 29 23:42:43.043239 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:43.043209    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-pcs7d"]
Oct 29 23:42:44.016967 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:44.016933    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0970c30-301a-420b-b7e4-87b7ad4200fc" path="/var/lib/kubelet/pods/b0970c30-301a-420b-b7e4-87b7ad4200fc/volumes"
Oct 29 23:42:47.481888 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:42:47.481828    6203 scope.go:122] "RemoveContainer" containerID="f4de73d10c12834de0ae9153ed621e62e5a0bb5a5e6f6c8d54393b8bce614214"
Oct 29 23:43:09.923563 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:43:09.923524    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:43:11.033232 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:11.033202    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"]
Oct 29 23:43:11.047661 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:11.047632    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4wj4m"]
Oct 29 23:43:12.016703 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:12.016667    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2207d0a2-2213-45a0-9fdf-c4666729bea7" path="/var/lib/kubelet/pods/2207d0a2-2213-45a0-9fdf-c4666729bea7/volumes"
Oct 29 23:43:21.030375 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:21.030340    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"]
Oct 29 23:43:21.044916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:21.044871    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sqv5r"]
Oct 29 23:43:22.016564 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:22.016533    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1b2762-bd22-4252-8c76-af6004a0b947" path="/var/lib/kubelet/pods/3d1b2762-bd22-4252-8c76-af6004a0b947/volumes"
Oct 29 23:43:39.923413 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:43:39.923369    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:43:46.032396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:46.032362    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"]
Oct 29 23:43:46.046801 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:46.046771    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-fdxbp"]
Oct 29 23:43:47.520300 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:47.520269    6203 scope.go:122] "RemoveContainer" containerID="8e4364942cc9a21d8d3307b3d61e433763680dd878644e11aac825b49303303a"
Oct 29 23:43:47.529072 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:47.529051    6203 scope.go:122] "RemoveContainer" containerID="98bc93739a72ede8714404d8b085dfa19e5299cd17372fa11d1b971750906a5a"
Oct 29 23:43:47.537006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:47.536985    6203 scope.go:122] "RemoveContainer" containerID="204e737c00f71d856d973387690f0665255d4a452067132dd4d5d4cef7c5903f"
Oct 29 23:43:48.017031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:43:48.016993    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afa91ff-e30d-4e44-92a7-85f0af972c45" path="/var/lib/kubelet/pods/9afa91ff-e30d-4e44-92a7-85f0af972c45/volumes"
Oct 29 23:44:03.411093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:03.411005    6203 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-746ff8f49f-vqnqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:44:03.411093 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:03.411072    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-746ff8f49f-vqnqf" podUID="bf9e8ad1-a722-4619-94b6-32d6e97002fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:44:04.278158 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278123    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"]
Oct 29 23:44:04.278505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278491    6203 cpu_manager.go:414] "containerMap: removing container" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy"
Oct 29 23:44:04.278557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278506    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy"
Oct 29 23:44:04.278557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278523    6203 cpu_manager.go:414] "containerMap: removing container" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy"
Oct 29 23:44:04.278557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278528    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy"
Oct 29 23:44:04.278557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278540    6203 cpu_manager.go:414] "containerMap: removing container" podUID="746c044c-105d-4d84-bb5d-e16ba4c3cdce" containerName="revoke-keys"
Oct 29 23:44:04.278557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278546    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746c044c-105d-4d84-bb5d-e16ba4c3cdce" containerName="revoke-keys"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278566    6203 cpu_manager.go:414] "containerMap: removing container" podUID="50eaa5fa-65c7-407d-9662-e59caabdf799" containerName="payload-pre-processing"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278572    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50eaa5fa-65c7-407d-9662-e59caabdf799" containerName="payload-pre-processing"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278580    6203 cpu_manager.go:414] "containerMap: removing container" podUID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" containerName="payload-processing"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278585    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" containerName="payload-processing"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278646    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278654    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2788a29-47c2-42e8-8672-8a789bf705ad" containerName="istio-proxy"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278660    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="746c044c-105d-4d84-bb5d-e16ba4c3cdce" containerName="revoke-keys"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278668    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="50eaa5fa-65c7-407d-9662-e59caabdf799" containerName="payload-pre-processing"
Oct 29 23:44:04.278715 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.278673    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7dd383-14cd-4cb6-9942-7b59bc74413b" containerName="payload-processing"
Oct 29 23:44:04.283791 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.283774    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.302296 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.302269    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2qkd\" (UniqueName: \"kubernetes.io/projected/23798fc1-6abb-43b3-b20e-ef3e814ece08-kube-api-access-d2qkd\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.302450 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.302335    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.302495 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.302459    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.302612 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.302591    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"]
Oct 29 23:44:04.315354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.315330    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:44:04.315354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.315346    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:44:04.315531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.315410    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:44:04.315531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.315418    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:44:04.403622 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.403587    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2qkd\" (UniqueName: \"kubernetes.io/projected/23798fc1-6abb-43b3-b20e-ef3e814ece08-kube-api-access-d2qkd\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.403819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.403647    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.403819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.403682    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.404322 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.404300    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.406088 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.406064    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.429995 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.429963    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2qkd\" (UniqueName: \"kubernetes.io/projected/23798fc1-6abb-43b3-b20e-ef3e814ece08-kube-api-access-d2qkd\") pod \"maas-api-revoke-sub-d576e68b-8btxv\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.596617 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.596533    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:04.896688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:04.896655    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"]
Oct 29 23:44:04.898158 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:44:04.898129    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23798fc1_6abb_43b3_b20e_ef3e814ece08.slice/crio-9480d84dda76bd9ef7b6c7604a6cd1e7ca583aa47e8614c5430de8ad70694b28 WatchSource:0}: Error finding container 9480d84dda76bd9ef7b6c7604a6cd1e7ca583aa47e8614c5430de8ad70694b28: Status 404 returned error can't find the container with id 9480d84dda76bd9ef7b6c7604a6cd1e7ca583aa47e8614c5430de8ad70694b28
Oct 29 23:44:05.607712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:05.607673    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv" event={"ID":"23798fc1-6abb-43b3-b20e-ef3e814ece08","Type":"ContainerStarted","Data":"75d6edff1d00f737fb1ccf6fd2a6b5f4a42a715c18d08c8dc846d9eee9fea200"}
Oct 29 23:44:05.608158 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:05.607742    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv" event={"ID":"23798fc1-6abb-43b3-b20e-ef3e814ece08","Type":"ContainerStarted","Data":"9480d84dda76bd9ef7b6c7604a6cd1e7ca583aa47e8614c5430de8ad70694b28"}
Oct 29 23:44:05.653541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:05.653484    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv" podStartSLOduration=1.653470029 podStartE2EDuration="1.653470029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:05.652686613 +0000 UTC m=+1823.756946999" watchObservedRunningTime="2026-10-29 23:44:05.653470029 +0000 UTC m=+1823.757730447"
Oct 29 23:44:06.614070 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:06.614029    6203 generic.go:356] "Generic (PLEG): container finished" podID="23798fc1-6abb-43b3-b20e-ef3e814ece08" containerID="75d6edff1d00f737fb1ccf6fd2a6b5f4a42a715c18d08c8dc846d9eee9fea200" exitCode=0
Oct 29 23:44:06.614464 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:06.614079    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv" event={"ID":"23798fc1-6abb-43b3-b20e-ef3e814ece08","Type":"ContainerDied","Data":"75d6edff1d00f737fb1ccf6fd2a6b5f4a42a715c18d08c8dc846d9eee9fea200"}
Oct 29 23:44:07.028316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.028277    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"]
Oct 29 23:44:07.031048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.031023    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-nl2dj"]
Oct 29 23:44:07.751231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.751208    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:07.831693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.831666    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23798fc1-6abb-43b3-b20e-ef3e814ece08-kube-api-access-d2qkd\" (UniqueName: \"kubernetes.io/projected/23798fc1-6abb-43b3-b20e-ef3e814ece08-kube-api-access-d2qkd\") pod \"23798fc1-6abb-43b3-b20e-ef3e814ece08\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") "
Oct 29 23:44:07.831862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.831783    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-openshift-ca\") pod \"23798fc1-6abb-43b3-b20e-ef3e814ece08\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") "
Oct 29 23:44:07.831862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.831842    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-xks-ca\") pod \"23798fc1-6abb-43b3-b20e-ef3e814ece08\" (UID: \"23798fc1-6abb-43b3-b20e-ef3e814ece08\") "
Oct 29 23:44:07.832173 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.832146    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-openshift-ca" pod "23798fc1-6abb-43b3-b20e-ef3e814ece08" (UID: "23798fc1-6abb-43b3-b20e-ef3e814ece08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:44:07.833720 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.833697    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23798fc1-6abb-43b3-b20e-ef3e814ece08-kube-api-access-d2qkd" pod "23798fc1-6abb-43b3-b20e-ef3e814ece08" (UID: "23798fc1-6abb-43b3-b20e-ef3e814ece08"). InnerVolumeSpecName "kube-api-access-d2qkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:44:07.833822 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.833787    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-xks-ca" pod "23798fc1-6abb-43b3-b20e-ef3e814ece08" (UID: "23798fc1-6abb-43b3-b20e-ef3e814ece08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:44:07.932901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.932852    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:44:07.933047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.932909    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2qkd\" (UniqueName: \"kubernetes.io/projected/23798fc1-6abb-43b3-b20e-ef3e814ece08-kube-api-access-d2qkd\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:44:07.933047 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:07.932926    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23798fc1-6abb-43b3-b20e-ef3e814ece08-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:44:08.016657 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:08.016622    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58edd9f-3da2-438e-9862-1d3982de629e" path="/var/lib/kubelet/pods/d58edd9f-3da2-438e-9862-1d3982de629e/volumes"
Oct 29 23:44:08.624789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:08.624759    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"
Oct 29 23:44:08.624789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:08.624777    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv" event={"ID":"23798fc1-6abb-43b3-b20e-ef3e814ece08","Type":"ContainerDied","Data":"9480d84dda76bd9ef7b6c7604a6cd1e7ca583aa47e8614c5430de8ad70694b28"}
Oct 29 23:44:08.625034 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:08.624822    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9480d84dda76bd9ef7b6c7604a6cd1e7ca583aa47e8614c5430de8ad70694b28"
Oct 29 23:44:09.923704 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:44:09.923668    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:44:20.825829 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.825785    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"]
Oct 29 23:44:20.826478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.826372    6203 cpu_manager.go:414] "containerMap: removing container" podUID="23798fc1-6abb-43b3-b20e-ef3e814ece08" containerName="revoke-keys"
Oct 29 23:44:20.826478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.826387    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23798fc1-6abb-43b3-b20e-ef3e814ece08" containerName="revoke-keys"
Oct 29 23:44:20.826478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.826461    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="23798fc1-6abb-43b3-b20e-ef3e814ece08" containerName="revoke-keys"
Oct 29 23:44:20.837297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.837260    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:20.849376 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.849345    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"]
Oct 29 23:44:20.854952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.854925    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:44:20.855083 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.854998    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:44:20.861667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.861643    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:44:20.861826 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.861810    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:44:20.999731 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.999695    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhdb\" (UniqueName: \"kubernetes.io/projected/04177281-b5d0-4300-b441-363f2e175ce6-kube-api-access-lnhdb\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:20.999963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.999755    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:20.999963 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:20.999910    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.101336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.101260    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhdb\" (UniqueName: \"kubernetes.io/projected/04177281-b5d0-4300-b441-363f2e175ce6-kube-api-access-lnhdb\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.101336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.101305    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.101530 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.101347    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.102156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.102136    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.103679 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.103659    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.127623 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.127599    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhdb\" (UniqueName: \"kubernetes.io/projected/04177281-b5d0-4300-b441-363f2e175ce6-kube-api-access-lnhdb\") pod \"maas-api-revoke-sub-d2bc93bc-7vzrp\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.153528 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.153500    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:21.451751 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.451726    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"]
Oct 29 23:44:21.453936 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:44:21.453872    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04177281_b5d0_4300_b441_363f2e175ce6.slice/crio-d642880e05a5ed2845f2db8fe23d3705fd803abaaf966596922248445fd32280 WatchSource:0}: Error finding container d642880e05a5ed2845f2db8fe23d3705fd803abaaf966596922248445fd32280: Status 404 returned error can't find the container with id d642880e05a5ed2845f2db8fe23d3705fd803abaaf966596922248445fd32280
Oct 29 23:44:21.701364 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:21.701279    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp" event={"ID":"04177281-b5d0-4300-b441-363f2e175ce6","Type":"ContainerStarted","Data":"d642880e05a5ed2845f2db8fe23d3705fd803abaaf966596922248445fd32280"}
Oct 29 23:44:22.708033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:22.707990    6203 generic.go:356] "Generic (PLEG): container finished" podID="04177281-b5d0-4300-b441-363f2e175ce6" containerID="48b699b1167e6042dfd2e7571fcec8c9b07ab929b17e320fc71ed454f81437c5" exitCode=0
Oct 29 23:44:22.708496 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:22.708077    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp" event={"ID":"04177281-b5d0-4300-b441-363f2e175ce6","Type":"ContainerDied","Data":"48b699b1167e6042dfd2e7571fcec8c9b07ab929b17e320fc71ed454f81437c5"}
Oct 29 23:44:23.843542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:23.843514    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:24.032302 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.032274    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-xks-ca\") pod \"04177281-b5d0-4300-b441-363f2e175ce6\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") "
Oct 29 23:44:24.032468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.032315    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04177281-b5d0-4300-b441-363f2e175ce6-kube-api-access-lnhdb\" (UniqueName: \"kubernetes.io/projected/04177281-b5d0-4300-b441-363f2e175ce6-kube-api-access-lnhdb\") pod \"04177281-b5d0-4300-b441-363f2e175ce6\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") "
Oct 29 23:44:24.032468 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.032345    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-openshift-ca\") pod \"04177281-b5d0-4300-b441-363f2e175ce6\" (UID: \"04177281-b5d0-4300-b441-363f2e175ce6\") "
Oct 29 23:44:24.032750 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.032727    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-openshift-ca" pod "04177281-b5d0-4300-b441-363f2e175ce6" (UID: "04177281-b5d0-4300-b441-363f2e175ce6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:44:24.034398 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.034369    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-xks-ca" pod "04177281-b5d0-4300-b441-363f2e175ce6" (UID: "04177281-b5d0-4300-b441-363f2e175ce6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:44:24.034511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.034416    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04177281-b5d0-4300-b441-363f2e175ce6-kube-api-access-lnhdb" pod "04177281-b5d0-4300-b441-363f2e175ce6" (UID: "04177281-b5d0-4300-b441-363f2e175ce6"). InnerVolumeSpecName "kube-api-access-lnhdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:44:24.133757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.133661    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:44:24.133757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.133708    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhdb\" (UniqueName: \"kubernetes.io/projected/04177281-b5d0-4300-b441-363f2e175ce6-kube-api-access-lnhdb\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:44:24.133757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.133718    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04177281-b5d0-4300-b441-363f2e175ce6-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:44:24.718727 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.718696    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"
Oct 29 23:44:24.718951 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.718691    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp" event={"ID":"04177281-b5d0-4300-b441-363f2e175ce6","Type":"ContainerDied","Data":"d642880e05a5ed2845f2db8fe23d3705fd803abaaf966596922248445fd32280"}
Oct 29 23:44:24.718951 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:24.718828    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d642880e05a5ed2845f2db8fe23d3705fd803abaaf966596922248445fd32280"
Oct 29 23:44:31.028679 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:31.028644    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"]
Oct 29 23:44:31.043042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:31.043010    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-dms6z"]
Oct 29 23:44:32.016688 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.016653    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87" path="/var/lib/kubelet/pods/3dc4bbe6-ff64-4e2b-a8e6-fa8fa9f6aa87/volumes"
Oct 29 23:44:32.949351 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.949315    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"]
Oct 29 23:44:32.950167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.950135    6203 cpu_manager.go:414] "containerMap: removing container" podUID="04177281-b5d0-4300-b441-363f2e175ce6" containerName="revoke-keys"
Oct 29 23:44:32.950167 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.950159    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04177281-b5d0-4300-b441-363f2e175ce6" containerName="revoke-keys"
Oct 29 23:44:32.950467 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.950377    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="04177281-b5d0-4300-b441-363f2e175ce6" containerName="revoke-keys"
Oct 29 23:44:32.954374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.954352    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:32.976964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.976924    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-06f91a-openshift-default-dockercfg-vqmp5\""
Oct 29 23:44:32.993985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:32.993957    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"]
Oct 29 23:44:33.009354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009327    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxxnc\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-kube-api-access-pxxnc\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009361    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-socket\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009386    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-certs\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009477    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-credential-socket\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009503    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1361d43d-1046-4112-80da-c54ee34d5972-istiod-ca-cert\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009522    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-data\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009656 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009585    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-envoy\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009694 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009667    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1361d43d-1046-4112-80da-c54ee34d5972-istio-podinfo\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.009728 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.009697    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-istio-token\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110086    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-istio-token\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110162    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxxnc\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-kube-api-access-pxxnc\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110193    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-socket\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110229    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-certs\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110289 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110273    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-credential-socket\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110303    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1361d43d-1046-4112-80da-c54ee34d5972-istiod-ca-cert\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110341    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-data\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110382    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-envoy\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110516 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110447    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1361d43d-1046-4112-80da-c54ee34d5972-istio-podinfo\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110688    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-socket\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.110961 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110930    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-certs\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.111125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.110992    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-credential-socket\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.111125 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.111066    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-data\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.111237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.111139    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1361d43d-1046-4112-80da-c54ee34d5972-istiod-ca-cert\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.112903 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.112864    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-envoy\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.113169 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.113150    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1361d43d-1046-4112-80da-c54ee34d5972-istio-podinfo\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.136716 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.136690    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxxnc\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-kube-api-access-pxxnc\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.137062 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.137046    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-istio-token\") pod \"e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") " pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.267759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.267670    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:33.584298 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.584274    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"]
Oct 29 23:44:33.585540 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:44:33.585514    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1361d43d_1046_4112_80da_c54ee34d5972.slice/crio-80453962f19627a5e9dbadaf73e2d63116f505425bb03096205b967c3a86057b WatchSource:0}: Error finding container 80453962f19627a5e9dbadaf73e2d63116f505425bb03096205b967c3a86057b: Status 404 returned error can't find the container with id 80453962f19627a5e9dbadaf73e2d63116f505425bb03096205b967c3a86057b
Oct 29 23:44:33.587595 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.587560    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:44:33.587704 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.587626    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:44:33.587704 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.587654    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:44:33.771249 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.771212    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" event={"ID":"1361d43d-1046-4112-80da-c54ee34d5972","Type":"ContainerStarted","Data":"a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4"}
Oct 29 23:44:33.771444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.771277    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" event={"ID":"1361d43d-1046-4112-80da-c54ee34d5972","Type":"ContainerStarted","Data":"80453962f19627a5e9dbadaf73e2d63116f505425bb03096205b967c3a86057b"}
Oct 29 23:44:33.822442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:33.822393    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podStartSLOduration=1.822379711 podStartE2EDuration="1.822379711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:33.82105831 +0000 UTC m=+1851.925318696" watchObservedRunningTime="2026-10-29 23:44:33.822379711 +0000 UTC m=+1851.926640096"
Oct 29 23:44:34.268349 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:34.268317    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:34.273670 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:34.273645    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:34.776491 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:34.776455    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:34.777664 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:34.777642    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:38.018658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.018629    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"]
Oct 29 23:44:38.022107 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.022086    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.045607 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.045582    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"]
Oct 29 23:44:38.058152 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.058120    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-06f91a\""
Oct 29 23:44:38.059580 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.059550    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb6qb\" (UniqueName: \"kubernetes.io/projected/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-kube-api-access-pb6qb\") pod \"payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.059784 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.059758    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.082957 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.082931    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"]
Oct 29 23:44:38.086529 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.086506    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.106798 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.106765    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e09ed170-dockercfg-t5xnk\""
Oct 29 23:44:38.109004 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.108981    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"]
Oct 29 23:44:38.161055 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.161008    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb6qb\" (UniqueName: \"kubernetes.io/projected/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-kube-api-access-pb6qb\") pod \"payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.162033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.161117    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzkt\" (UniqueName: \"kubernetes.io/projected/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-kube-api-access-8lzkt\") pod \"payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.162033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.161196    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.162033 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.161233    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.162259 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.162224    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.187117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.187089    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb6qb\" (UniqueName: \"kubernetes.io/projected/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-kube-api-access-pb6qb\") pod \"payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.262073 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.262039    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzkt\" (UniqueName: \"kubernetes.io/projected/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-kube-api-access-8lzkt\") pod \"payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.262254 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.262099    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.262724 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.262694    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.288000 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.287934    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzkt\" (UniqueName: \"kubernetes.io/projected/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-kube-api-access-8lzkt\") pod \"payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.336596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.336567    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.398744 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.398587    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.636632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.636604    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"]
Oct 29 23:44:38.638150 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:44:38.638123    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fce1fa9_8e0e_4d3c_a9aa_84c3e2f57d60.slice/crio-fbdd353652f54da1e84ce5e5a9f623c1a3f8c8333ad7961d67e483fb9ebd397a WatchSource:0}: Error finding container fbdd353652f54da1e84ce5e5a9f623c1a3f8c8333ad7961d67e483fb9ebd397a: Status 404 returned error can't find the container with id fbdd353652f54da1e84ce5e5a9f623c1a3f8c8333ad7961d67e483fb9ebd397a
Oct 29 23:44:38.714760 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.714732    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"]
Oct 29 23:44:38.716271 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:44:38.716246    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6f1e95_3d0d_42d9_84b5_cc169c1e06e9.slice/crio-69a0798fe989f1a7c396a7b3c59e4908bea6bdb45f75a1ca018fcd3208b8fdc3 WatchSource:0}: Error finding container 69a0798fe989f1a7c396a7b3c59e4908bea6bdb45f75a1ca018fcd3208b8fdc3: Status 404 returned error can't find the container with id 69a0798fe989f1a7c396a7b3c59e4908bea6bdb45f75a1ca018fcd3208b8fdc3
Oct 29 23:44:38.796797 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.796765    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" event={"ID":"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9","Type":"ContainerStarted","Data":"81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621"}
Oct 29 23:44:38.796983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.796833    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:38.796983 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.796859    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" event={"ID":"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9","Type":"ContainerStarted","Data":"69a0798fe989f1a7c396a7b3c59e4908bea6bdb45f75a1ca018fcd3208b8fdc3"}
Oct 29 23:44:38.798414 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.798390    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" event={"ID":"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60","Type":"ContainerStarted","Data":"16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081"}
Oct 29 23:44:38.798533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.798437    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" event={"ID":"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60","Type":"ContainerStarted","Data":"fbdd353652f54da1e84ce5e5a9f623c1a3f8c8333ad7961d67e483fb9ebd397a"}
Oct 29 23:44:38.798533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.798466    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:38.844131 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.844060    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" podStartSLOduration=0.844046083 podStartE2EDuration="844.046083ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:38.84318269 +0000 UTC m=+1856.947443078" watchObservedRunningTime="2026-10-29 23:44:38.844046083 +0000 UTC m=+1856.948306468"
Oct 29 23:44:38.889236 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:38.889123    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" podStartSLOduration=1.889111233 podStartE2EDuration="1.889111233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:38.888997535 +0000 UTC m=+1856.993257921" watchObservedRunningTime="2026-10-29 23:44:38.889111233 +0000 UTC m=+1856.993371619"
Oct 29 23:44:39.923576 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:44:39.923540    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:44:47.565948 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:47.565914    6203 scope.go:122] "RemoveContainer" containerID="7ee10302016a6d087b4aea1900c1ed167869b7679b4ea97c18389eb3dc844717"
Oct 29 23:44:47.574780 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:47.574760    6203 scope.go:122] "RemoveContainer" containerID="55d5b10076ee144249b942e02d06fa8a30aaed35e0d812296daf10544697d7db"
Oct 29 23:44:47.582543 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:47.582519    6203 scope.go:122] "RemoveContainer" containerID="c0aa50e3e907647b68bf2565de044380983b2ba26ce4d3f57ade01c9478312ee"
Oct 29 23:44:48.039186 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:48.039148    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"]
Oct 29 23:44:48.053373 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:48.053343    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-p9wdh"]
Oct 29 23:44:49.808024 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:49.807986    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:44:49.808475 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:49.808088    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:44:50.018451 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:50.018416    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2467e88d-1bb3-4840-bf1a-a7c16976eaa9" path="/var/lib/kubelet/pods/2467e88d-1bb3-4840-bf1a-a7c16976eaa9/volumes"
Oct 29 23:44:50.818036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:50.818004    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:44:50.834244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:50.818077    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:44:51.899635 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:51.899596    6203 generic.go:356] "Generic (PLEG): container finished" podID="1361d43d-1046-4112-80da-c54ee34d5972" containerID="a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4" exitCode=0
Oct 29 23:44:51.900188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:51.899645    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" event={"ID":"1361d43d-1046-4112-80da-c54ee34d5972","Type":"ContainerDied","Data":"a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4"}
Oct 29 23:44:51.900188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:51.900142    6203 scope.go:122] "RemoveContainer" containerID="a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4"
Oct 29 23:44:51.901581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:51.901543    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:44:51.901689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:51.901627    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:44:51.901689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:51.901679    6203 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 29 23:44:52.907192 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:52.907160    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" event={"ID":"1361d43d-1046-4112-80da-c54ee34d5972","Type":"ContainerStarted","Data":"bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd"}
Oct 29 23:44:53.268143 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.268106    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:53.268327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.268281    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 29 23:44:53.268374 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.268334    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 29 23:44:53.892290 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.892252    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"]
Oct 29 23:44:53.896582 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.896560    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:53.902934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.902903    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"]
Oct 29 23:44:53.915353 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.915330    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-06f91a\"/\"default-dockercfg-8s9sg\""
Oct 29 23:44:53.915684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.915360    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-06f91a\"/\"simulator-single-main-06f91a-kserve-self-signed-certs\""
Oct 29 23:44:53.952981 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.950303    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-06f91a\"/\"openshift-service-ca.crt\""
Oct 29 23:44:53.952981 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:53.950431    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-06f91a\"/\"kube-root-ca.crt\""
Oct 29 23:44:54.063681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.063646    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-dshm\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.063681 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.063686    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-tmp-dir\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.063994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.063927    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-home\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.063994 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.063977    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac392465-60bf-403d-a15b-80eb7087da46-tls-certs\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.064185 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.064170    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-model-cache\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.064229 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.064203    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8n6b\" (UniqueName: \"kubernetes.io/projected/ac392465-60bf-403d-a15b-80eb7087da46-kube-api-access-w8n6b\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165265    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-home\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165357 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165321    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac392465-60bf-403d-a15b-80eb7087da46-tls-certs\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165395    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-model-cache\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165432    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8n6b\" (UniqueName: \"kubernetes.io/projected/ac392465-60bf-403d-a15b-80eb7087da46-kube-api-access-w8n6b\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165459    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-dshm\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165614 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165489    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-tmp-dir\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165745    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-home\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165791    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-model-cache\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.165985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.165950    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-tmp-dir\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.167858 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.167833    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-dshm\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.168180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.168162    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac392465-60bf-403d-a15b-80eb7087da46-tls-certs\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.201889 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.201843    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8n6b\" (UniqueName: \"kubernetes.io/projected/ac392465-60bf-403d-a15b-80eb7087da46-kube-api-access-w8n6b\") pod \"simulator-single-main-06f91a-kserve-555b666c95-zd2pf\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") " pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.210942 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.210920    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:54.268051 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.268019    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 29 23:44:54.268217 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.268082    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 29 23:44:54.548904 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.541037    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"]
Oct 29 23:44:54.559418 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:44:54.559388    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac392465_60bf_403d_a15b_80eb7087da46.slice/crio-a41846a66335ccabb4db4c4f7039b8f426e4a997cedbe9ffdad1566786629ee7 WatchSource:0}: Error finding container a41846a66335ccabb4db4c4f7039b8f426e4a997cedbe9ffdad1566786629ee7: Status 404 returned error can't find the container with id a41846a66335ccabb4db4c4f7039b8f426e4a997cedbe9ffdad1566786629ee7
Oct 29 23:44:54.922778 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.922741    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf" event={"ID":"ac392465-60bf-403d-a15b-80eb7087da46","Type":"ContainerStarted","Data":"819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a"}
Oct 29 23:44:54.923220 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:54.922791    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf" event={"ID":"ac392465-60bf-403d-a15b-80eb7087da46","Type":"ContainerStarted","Data":"a41846a66335ccabb4db4c4f7039b8f426e4a997cedbe9ffdad1566786629ee7"}
Oct 29 23:44:55.013025 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:55.012972    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf" podStartSLOduration=2.01295736 podStartE2EDuration="2.01295736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:44:55.012524191 +0000 UTC m=+1873.116784577" watchObservedRunningTime="2026-10-29 23:44:55.01295736 +0000 UTC m=+1873.117217745"
Oct 29 23:44:55.212018 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:55.211942    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:44:55.268459 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:55.268417    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 29 23:44:55.268662 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:55.268485    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 29 23:44:56.268800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:56.268758    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 29 23:44:56.269305 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:56.268831    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 29 23:44:57.268865 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:57.268830    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 29 23:44:57.269297 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:57.268917    6203 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 29 23:44:58.484049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:58.484013    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:58.484653 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:58.484459    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:44:58.486460 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:44:58.486439    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:45:00.151057 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.151021    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"]
Oct 29 23:45:00.154916 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.154893    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.174036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.174007    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"]
Oct 29 23:45:00.186574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.186537    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:45:00.186738 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.186596    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:45:00.186738 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.186608    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:45:00.216132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.216094    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zqb\" (UniqueName: \"kubernetes.io/projected/c62da78b-e1d0-4067-adab-f80a06354e5e-kube-api-access-87zqb\") pod \"maas-api-key-cleanup-29888625-sw5rf\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.216315 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.216159    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62da78b-e1d0-4067-adab-f80a06354e5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888625-sw5rf\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.316739 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.316706    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zqb\" (UniqueName: \"kubernetes.io/projected/c62da78b-e1d0-4067-adab-f80a06354e5e-kube-api-access-87zqb\") pod \"maas-api-key-cleanup-29888625-sw5rf\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.316985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.316751    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62da78b-e1d0-4067-adab-f80a06354e5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888625-sw5rf\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.317494 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.317475    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62da78b-e1d0-4067-adab-f80a06354e5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888625-sw5rf\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.342809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.342778    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zqb\" (UniqueName: \"kubernetes.io/projected/c62da78b-e1d0-4067-adab-f80a06354e5e-kube-api-access-87zqb\") pod \"maas-api-key-cleanup-29888625-sw5rf\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.467733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.467695    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:00.766005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.765977    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"]
Oct 29 23:45:00.767520 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:45:00.767493    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62da78b_e1d0_4067_adab_f80a06354e5e.slice/crio-94afa62af67b3448fe48d093777dc0a124a7b8d46b052617bcccb488c209add4 WatchSource:0}: Error finding container 94afa62af67b3448fe48d093777dc0a124a7b8d46b052617bcccb488c209add4: Status 404 returned error can't find the container with id 94afa62af67b3448fe48d093777dc0a124a7b8d46b052617bcccb488c209add4
Oct 29 23:45:00.980672 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.980635    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" event={"ID":"c62da78b-e1d0-4067-adab-f80a06354e5e","Type":"ContainerStarted","Data":"b4edd3600343521fa4263babdbadfbbb28c000b1701cf894c39e095dc1353d76"}
Oct 29 23:45:00.980815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:00.980704    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" event={"ID":"c62da78b-e1d0-4067-adab-f80a06354e5e","Type":"ContainerStarted","Data":"94afa62af67b3448fe48d093777dc0a124a7b8d46b052617bcccb488c209add4"}
Oct 29 23:45:01.029337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:01.029233    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" podStartSLOduration=1.029218928 podStartE2EDuration="1.029218928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:01.028486779 +0000 UTC m=+1879.132747167" watchObservedRunningTime="2026-10-29 23:45:01.029218928 +0000 UTC m=+1879.133479313"
Oct 29 23:45:01.986999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:01.986958    6203 generic.go:356] "Generic (PLEG): container finished" podID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerID="b4edd3600343521fa4263babdbadfbbb28c000b1701cf894c39e095dc1353d76" exitCode=60
Oct 29 23:45:01.987465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:01.987042    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" event={"ID":"c62da78b-e1d0-4067-adab-f80a06354e5e","Type":"ContainerDied","Data":"b4edd3600343521fa4263babdbadfbbb28c000b1701cf894c39e095dc1353d76"}
Oct 29 23:45:01.987465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:01.987266    6203 scope.go:122] "RemoveContainer" containerID="b4edd3600343521fa4263babdbadfbbb28c000b1701cf894c39e095dc1353d76"
Oct 29 23:45:02.994084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:02.993993    6203 generic.go:356] "Generic (PLEG): container finished" podID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerID="757e3a5114862e97aefcb5ae640d4e6c5d2258ce2f19a8c345b08a86f0351dc1" exitCode=60
Oct 29 23:45:02.994084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:02.994067    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" event={"ID":"c62da78b-e1d0-4067-adab-f80a06354e5e","Type":"ContainerDied","Data":"757e3a5114862e97aefcb5ae640d4e6c5d2258ce2f19a8c345b08a86f0351dc1"}
Oct 29 23:45:02.994581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:02.994125    6203 scope.go:122] "RemoveContainer" containerID="b4edd3600343521fa4263babdbadfbbb28c000b1701cf894c39e095dc1353d76"
Oct 29 23:45:02.994581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:02.994408    6203 scope.go:122] "RemoveContainer" containerID="757e3a5114862e97aefcb5ae640d4e6c5d2258ce2f19a8c345b08a86f0351dc1"
Oct 29 23:45:02.994734 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:02.994712    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888625-sw5rf_odh-ai-gateway-infra(c62da78b-e1d0-4067-adab-f80a06354e5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e"
Oct 29 23:45:04.001421 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:04.001394    6203 scope.go:122] "RemoveContainer" containerID="757e3a5114862e97aefcb5ae640d4e6c5d2258ce2f19a8c345b08a86f0351dc1"
Oct 29 23:45:04.001793 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:04.001632    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888625-sw5rf_odh-ai-gateway-infra(c62da78b-e1d0-4067-adab-f80a06354e5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e"
Oct 29 23:45:04.211454 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:04.211421    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:45:04.224131 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:04.224092    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:45:05.017356 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:05.017323    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:45:09.923696 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:09.923653    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:45:12.809394 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:12.809363    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"]
Oct 29 23:45:12.813144 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:12.813126    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:12.832517 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:12.832489    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:45:12.848810 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:12.848776    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"]
Oct 29 23:45:12.940150 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:12.940113    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:12.940317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:12.940184    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:12.940407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:12.940310    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5mh\" (UniqueName: \"kubernetes.io/projected/f3456cba-6951-4f2f-af14-b088ef79dd6c-kube-api-access-6h5mh\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.041544 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.041508    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.041746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.041570    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.041746 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.041625    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5mh\" (UniqueName: \"kubernetes.io/projected/f3456cba-6951-4f2f-af14-b088ef79dd6c-kube-api-access-6h5mh\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.042202 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.042176    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.043979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.043954    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.067320 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.067252    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5mh\" (UniqueName: \"kubernetes.io/projected/f3456cba-6951-4f2f-af14-b088ef79dd6c-kube-api-access-6h5mh\") pod \"maas-api-revoke-sub-48a6af36-5pmsj\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.125857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.125825    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:13.434384 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:13.434359    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"]
Oct 29 23:45:13.435386 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:45:13.435360    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3456cba_6951_4f2f_af14_b088ef79dd6c.slice/crio-f40e7d10165c7eace85dcf6bff1eb24bc3390a8f0f807d865ba206a4e9911191 WatchSource:0}: Error finding container f40e7d10165c7eace85dcf6bff1eb24bc3390a8f0f807d865ba206a4e9911191: Status 404 returned error can't find the container with id f40e7d10165c7eace85dcf6bff1eb24bc3390a8f0f807d865ba206a4e9911191
Oct 29 23:45:14.060604 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:14.060514    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj" event={"ID":"f3456cba-6951-4f2f-af14-b088ef79dd6c","Type":"ContainerStarted","Data":"ef6ed4f48c731d4fc3606d4cee6189be313ab591388fc5481f955cd8407178f2"}
Oct 29 23:45:14.061079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:14.060600    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj" event={"ID":"f3456cba-6951-4f2f-af14-b088ef79dd6c","Type":"ContainerStarted","Data":"f40e7d10165c7eace85dcf6bff1eb24bc3390a8f0f807d865ba206a4e9911191"}
Oct 29 23:45:14.113990 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:14.113939    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj" podStartSLOduration=2.113924883 podStartE2EDuration="2.113924883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:14.113042394 +0000 UTC m=+1892.217302780" watchObservedRunningTime="2026-10-29 23:45:14.113924883 +0000 UTC m=+1892.218185311"
Oct 29 23:45:15.067566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:15.067530    6203 generic.go:356] "Generic (PLEG): container finished" podID="f3456cba-6951-4f2f-af14-b088ef79dd6c" containerID="ef6ed4f48c731d4fc3606d4cee6189be313ab591388fc5481f955cd8407178f2" exitCode=0
Oct 29 23:45:15.068017 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:15.067617    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj" event={"ID":"f3456cba-6951-4f2f-af14-b088ef79dd6c","Type":"ContainerDied","Data":"ef6ed4f48c731d4fc3606d4cee6189be313ab591388fc5481f955cd8407178f2"}
Oct 29 23:45:16.011347 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.011318    6203 scope.go:122] "RemoveContainer" containerID="757e3a5114862e97aefcb5ae640d4e6c5d2258ce2f19a8c345b08a86f0351dc1"
Oct 29 23:45:16.255771 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.255722    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:16.375721 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.375689    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-openshift-ca\") pod \"f3456cba-6951-4f2f-af14-b088ef79dd6c\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") "
Oct 29 23:45:16.375914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.375833    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3456cba-6951-4f2f-af14-b088ef79dd6c-kube-api-access-6h5mh\" (UniqueName: \"kubernetes.io/projected/f3456cba-6951-4f2f-af14-b088ef79dd6c-kube-api-access-6h5mh\") pod \"f3456cba-6951-4f2f-af14-b088ef79dd6c\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") "
Oct 29 23:45:16.375914 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.375871    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-xks-ca\") pod \"f3456cba-6951-4f2f-af14-b088ef79dd6c\" (UID: \"f3456cba-6951-4f2f-af14-b088ef79dd6c\") "
Oct 29 23:45:16.376127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.376092    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-openshift-ca" pod "f3456cba-6951-4f2f-af14-b088ef79dd6c" (UID: "f3456cba-6951-4f2f-af14-b088ef79dd6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:16.378120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.378098    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-xks-ca" pod "f3456cba-6951-4f2f-af14-b088ef79dd6c" (UID: "f3456cba-6951-4f2f-af14-b088ef79dd6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:45:16.378120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.378106    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3456cba-6951-4f2f-af14-b088ef79dd6c-kube-api-access-6h5mh" pod "f3456cba-6951-4f2f-af14-b088ef79dd6c" (UID: "f3456cba-6951-4f2f-af14-b088ef79dd6c"). InnerVolumeSpecName "kube-api-access-6h5mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:16.476627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.476591    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5mh\" (UniqueName: \"kubernetes.io/projected/f3456cba-6951-4f2f-af14-b088ef79dd6c-kube-api-access-6h5mh\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:16.476627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.476620    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:16.476627 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:16.476630    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3456cba-6951-4f2f-af14-b088ef79dd6c-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:17.079496 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:17.079470    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"
Oct 29 23:45:17.079496 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:17.079480    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj" event={"ID":"f3456cba-6951-4f2f-af14-b088ef79dd6c","Type":"ContainerDied","Data":"f40e7d10165c7eace85dcf6bff1eb24bc3390a8f0f807d865ba206a4e9911191"}
Oct 29 23:45:17.079717 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:17.079528    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f40e7d10165c7eace85dcf6bff1eb24bc3390a8f0f807d865ba206a4e9911191"
Oct 29 23:45:17.081665 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:17.081634    6203 generic.go:356] "Generic (PLEG): container finished" podID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerID="b7f90df74890cd9e7426070a0dfab53df40599b95f2a691d6d9effc244046163" exitCode=60
Oct 29 23:45:17.081832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:17.081705    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" event={"ID":"c62da78b-e1d0-4067-adab-f80a06354e5e","Type":"ContainerDied","Data":"b7f90df74890cd9e7426070a0dfab53df40599b95f2a691d6d9effc244046163"}
Oct 29 23:45:17.081832 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:17.081752    6203 scope.go:122] "RemoveContainer" containerID="757e3a5114862e97aefcb5ae640d4e6c5d2258ce2f19a8c345b08a86f0351dc1"
Oct 29 23:45:17.082129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:17.082112    6203 scope.go:122] "RemoveContainer" containerID="b7f90df74890cd9e7426070a0dfab53df40599b95f2a691d6d9effc244046163"
Oct 29 23:45:17.082375 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:17.082358    6203 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888625-sw5rf_odh-ai-gateway-infra(c62da78b-e1d0-4067-adab-f80a06354e5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e"
Oct 29 23:45:18.139702 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.139668    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"]
Oct 29 23:45:18.279453 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.279428    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:18.397159 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.397075    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62da78b-e1d0-4067-adab-f80a06354e5e-kube-api-access-87zqb\" (UniqueName: \"kubernetes.io/projected/c62da78b-e1d0-4067-adab-f80a06354e5e-kube-api-access-87zqb\") pod \"c62da78b-e1d0-4067-adab-f80a06354e5e\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") "
Oct 29 23:45:18.397159 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.397111    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c62da78b-e1d0-4067-adab-f80a06354e5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62da78b-e1d0-4067-adab-f80a06354e5e-maas-api-service-ca\") pod \"c62da78b-e1d0-4067-adab-f80a06354e5e\" (UID: \"c62da78b-e1d0-4067-adab-f80a06354e5e\") "
Oct 29 23:45:18.397495 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.397472    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62da78b-e1d0-4067-adab-f80a06354e5e-maas-api-service-ca" pod "c62da78b-e1d0-4067-adab-f80a06354e5e" (UID: "c62da78b-e1d0-4067-adab-f80a06354e5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:18.399268 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.399241    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62da78b-e1d0-4067-adab-f80a06354e5e-kube-api-access-87zqb" pod "c62da78b-e1d0-4067-adab-f80a06354e5e" (UID: "c62da78b-e1d0-4067-adab-f80a06354e5e"). InnerVolumeSpecName "kube-api-access-87zqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:18.498590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.498554    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87zqb\" (UniqueName: \"kubernetes.io/projected/c62da78b-e1d0-4067-adab-f80a06354e5e-kube-api-access-87zqb\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:18.498590 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:18.498583    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c62da78b-e1d0-4067-adab-f80a06354e5e-maas-api-service-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:19.094650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:19.094618    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"
Oct 29 23:45:19.094650 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:19.094647    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf" event={"ID":"c62da78b-e1d0-4067-adab-f80a06354e5e","Type":"ContainerDied","Data":"94afa62af67b3448fe48d093777dc0a124a7b8d46b052617bcccb488c209add4"}
Oct 29 23:45:19.094922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:19.094705    6203 scope.go:122] "RemoveContainer" containerID="b7f90df74890cd9e7426070a0dfab53df40599b95f2a691d6d9effc244046163"
Oct 29 23:45:19.152244 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:19.152207    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"]
Oct 29 23:45:19.166631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:19.166599    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888625-sw5rf"]
Oct 29 23:45:20.018161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:20.018124    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" path="/var/lib/kubelet/pods/c62da78b-e1d0-4067-adab-f80a06354e5e/volumes"
Oct 29 23:45:23.752231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752191    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"]
Oct 29 23:45:23.752666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752627    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:23.752666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752639    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:23.752666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752660    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:23.752666 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752664    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:23.752808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752677    6203 cpu_manager.go:414] "containerMap: removing container" podUID="f3456cba-6951-4f2f-af14-b088ef79dd6c" containerName="revoke-keys"
Oct 29 23:45:23.752808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752682    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3456cba-6951-4f2f-af14-b088ef79dd6c" containerName="revoke-keys"
Oct 29 23:45:23.752808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752739    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:23.752808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752750    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:23.752808 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.752759    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3456cba-6951-4f2f-af14-b088ef79dd6c" containerName="revoke-keys"
Oct 29 23:45:23.757190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.757173    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.776629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.776603    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"]
Oct 29 23:45:23.789038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.788956    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:45:23.789038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.788997    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:45:23.789210 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.789038    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:45:23.791599 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.791580    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:45:23.850285 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.850235    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.850490 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.850297    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.850490 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.850400    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkk9s\" (UniqueName: \"kubernetes.io/projected/59e2c046-6d89-4cf0-a340-94d81eb5eac6-kube-api-access-mkk9s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.951478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.951401    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.951478 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.951459    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.951706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.951517    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkk9s\" (UniqueName: \"kubernetes.io/projected/59e2c046-6d89-4cf0-a340-94d81eb5eac6-kube-api-access-mkk9s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.952197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.952174    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.953932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.953909    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:23.992757 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:23.992733    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkk9s\" (UniqueName: \"kubernetes.io/projected/59e2c046-6d89-4cf0-a340-94d81eb5eac6-kube-api-access-mkk9s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:24.069934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:24.069838    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:24.368215 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:24.368188    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"]
Oct 29 23:45:24.369946 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:45:24.369912    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e2c046_6d89_4cf0_a340_94d81eb5eac6.slice/crio-8f7b48befa4412146728dbcb7d6b65f423bd2519af91d751d7d661861e2fa96c WatchSource:0}: Error finding container 8f7b48befa4412146728dbcb7d6b65f423bd2519af91d751d7d661861e2fa96c: Status 404 returned error can't find the container with id 8f7b48befa4412146728dbcb7d6b65f423bd2519af91d751d7d661861e2fa96c
Oct 29 23:45:25.137675 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:25.137630    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g" event={"ID":"59e2c046-6d89-4cf0-a340-94d81eb5eac6","Type":"ContainerStarted","Data":"2255c630ec475ffc48d5a7a96e6cdac209057c053aeef2aa53a07e69918e1c37"}
Oct 29 23:45:25.138168 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:25.137732    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g" event={"ID":"59e2c046-6d89-4cf0-a340-94d81eb5eac6","Type":"ContainerStarted","Data":"8f7b48befa4412146728dbcb7d6b65f423bd2519af91d751d7d661861e2fa96c"}
Oct 29 23:45:25.188803 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:25.188743    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g" podStartSLOduration=2.188728469 podStartE2EDuration="2.188728469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:25.187323438 +0000 UTC m=+1903.291583825" watchObservedRunningTime="2026-10-29 23:45:25.188728469 +0000 UTC m=+1903.292988834"
Oct 29 23:45:26.145263 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:26.145219    6203 generic.go:356] "Generic (PLEG): container finished" podID="59e2c046-6d89-4cf0-a340-94d81eb5eac6" containerID="2255c630ec475ffc48d5a7a96e6cdac209057c053aeef2aa53a07e69918e1c37" exitCode=0
Oct 29 23:45:26.145667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:26.145304    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g" event={"ID":"59e2c046-6d89-4cf0-a340-94d81eb5eac6","Type":"ContainerDied","Data":"2255c630ec475ffc48d5a7a96e6cdac209057c053aeef2aa53a07e69918e1c37"}
Oct 29 23:45:27.281706 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.281680    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:27.387565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.387531    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-openshift-ca\") pod \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") "
Oct 29 23:45:27.387758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.387570    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-xks-ca\") pod \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") "
Oct 29 23:45:27.387758 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.387618    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e2c046-6d89-4cf0-a340-94d81eb5eac6-kube-api-access-mkk9s\" (UniqueName: \"kubernetes.io/projected/59e2c046-6d89-4cf0-a340-94d81eb5eac6-kube-api-access-mkk9s\") pod \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\" (UID: \"59e2c046-6d89-4cf0-a340-94d81eb5eac6\") "
Oct 29 23:45:27.387920 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.387893    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-openshift-ca" pod "59e2c046-6d89-4cf0-a340-94d81eb5eac6" (UID: "59e2c046-6d89-4cf0-a340-94d81eb5eac6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:27.389765 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.389739    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-xks-ca" pod "59e2c046-6d89-4cf0-a340-94d81eb5eac6" (UID: "59e2c046-6d89-4cf0-a340-94d81eb5eac6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:45:27.389859 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.389760    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e2c046-6d89-4cf0-a340-94d81eb5eac6-kube-api-access-mkk9s" pod "59e2c046-6d89-4cf0-a340-94d81eb5eac6" (UID: "59e2c046-6d89-4cf0-a340-94d81eb5eac6"). InnerVolumeSpecName "kube-api-access-mkk9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:27.488393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.488357    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:27.488393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.488388    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e2c046-6d89-4cf0-a340-94d81eb5eac6-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:27.488393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:27.488397    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkk9s\" (UniqueName: \"kubernetes.io/projected/59e2c046-6d89-4cf0-a340-94d81eb5eac6-kube-api-access-mkk9s\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:28.157162 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:28.157117    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g" event={"ID":"59e2c046-6d89-4cf0-a340-94d81eb5eac6","Type":"ContainerDied","Data":"8f7b48befa4412146728dbcb7d6b65f423bd2519af91d751d7d661861e2fa96c"}
Oct 29 23:45:28.157162 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:28.157168    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7b48befa4412146728dbcb7d6b65f423bd2519af91d751d7d661861e2fa96c"
Oct 29 23:45:28.157378 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:28.157128    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"
Oct 29 23:45:28.546371 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:28.546337    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"]
Oct 29 23:45:34.256206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:34.256168    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"]
Oct 29 23:45:34.289426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:34.289385    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"]
Oct 29 23:45:35.723772 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:35.723738    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"]
Oct 29 23:45:35.737798 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:35.737766    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-06f91a-t9v4g"]
Oct 29 23:45:36.018237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:36.018148    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e2c046-6d89-4cf0-a340-94d81eb5eac6" path="/var/lib/kubelet/pods/59e2c046-6d89-4cf0-a340-94d81eb5eac6/volumes"
Oct 29 23:45:39.257415 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:39.257372    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" podUID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" containerName="payload-processing" containerID="cri-o://16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081" gracePeriod=25
Oct 29 23:45:39.290075 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:39.289999    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" podUID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" containerName="payload-pre-processing" containerID="cri-o://81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621" gracePeriod=25
Oct 29 23:45:39.806667 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:39.806632    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" podUID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:39.806834 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:39.806671    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" podUID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:39.923548 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:39.923506    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:45:42.776394 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:42.776360    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"]
Oct 29 23:45:42.776934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:42.776601    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" containerID="cri-o://bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd" gracePeriod=30
Oct 29 23:45:43.486960 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.486924    6203 patch_prober.go:29] interesting pod/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:45:43.487151 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.486993    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:45:43.566441 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.566377    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf" podUID="ac392465-60bf-403d-a15b-80eb7087da46" containerName="main" containerID="cri-o://819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a" gracePeriod=45
Oct 29 23:45:43.827532 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.827505    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:45:43.849105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849062    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-tmp-dir\") pod \"ac392465-60bf-403d-a15b-80eb7087da46\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") "
Oct 29 23:45:43.849299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849115    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac392465-60bf-403d-a15b-80eb7087da46-kube-api-access-w8n6b\" (UniqueName: \"kubernetes.io/projected/ac392465-60bf-403d-a15b-80eb7087da46-kube-api-access-w8n6b\") pod \"ac392465-60bf-403d-a15b-80eb7087da46\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") "
Oct 29 23:45:43.849299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849146    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-model-cache\") pod \"ac392465-60bf-403d-a15b-80eb7087da46\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") "
Oct 29 23:45:43.849299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849174    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-dshm\") pod \"ac392465-60bf-403d-a15b-80eb7087da46\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") "
Oct 29 23:45:43.849299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849221    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac392465-60bf-403d-a15b-80eb7087da46-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac392465-60bf-403d-a15b-80eb7087da46-tls-certs\") pod \"ac392465-60bf-403d-a15b-80eb7087da46\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") "
Oct 29 23:45:43.849299 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849256    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-home\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-home\") pod \"ac392465-60bf-403d-a15b-80eb7087da46\" (UID: \"ac392465-60bf-403d-a15b-80eb7087da46\") "
Oct 29 23:45:43.849568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849394    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-tmp-dir" pod "ac392465-60bf-403d-a15b-80eb7087da46" (UID: "ac392465-60bf-403d-a15b-80eb7087da46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:43.849568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849510    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-home" pod "ac392465-60bf-403d-a15b-80eb7087da46" (UID: "ac392465-60bf-403d-a15b-80eb7087da46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:43.849568 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849524    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-model-cache" pod "ac392465-60bf-403d-a15b-80eb7087da46" (UID: "ac392465-60bf-403d-a15b-80eb7087da46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:43.849733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849709    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:43.849733 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849727    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:43.849844 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.849740    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:43.851461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.851434    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-dshm" pod "ac392465-60bf-403d-a15b-80eb7087da46" (UID: "ac392465-60bf-403d-a15b-80eb7087da46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:43.851572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.851468    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac392465-60bf-403d-a15b-80eb7087da46-tls-certs" pod "ac392465-60bf-403d-a15b-80eb7087da46" (UID: "ac392465-60bf-403d-a15b-80eb7087da46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:45:43.851572 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.851492    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac392465-60bf-403d-a15b-80eb7087da46-kube-api-access-w8n6b" pod "ac392465-60bf-403d-a15b-80eb7087da46" (UID: "ac392465-60bf-403d-a15b-80eb7087da46"). InnerVolumeSpecName "kube-api-access-w8n6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:43.950332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.950293    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8n6b\" (UniqueName: \"kubernetes.io/projected/ac392465-60bf-403d-a15b-80eb7087da46-kube-api-access-w8n6b\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:43.950332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.950326    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac392465-60bf-403d-a15b-80eb7087da46-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:43.950332 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:43.950336    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac392465-60bf-403d-a15b-80eb7087da46-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:44.241021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.240985    6203 generic.go:356] "Generic (PLEG): container finished" podID="ac392465-60bf-403d-a15b-80eb7087da46" containerID="819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a" exitCode=0
Oct 29 23:45:44.241345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.241036    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"
Oct 29 23:45:44.241345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.241066    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf" event={"ID":"ac392465-60bf-403d-a15b-80eb7087da46","Type":"ContainerDied","Data":"819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a"}
Oct 29 23:45:44.241345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.241117    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf" event={"ID":"ac392465-60bf-403d-a15b-80eb7087da46","Type":"ContainerDied","Data":"a41846a66335ccabb4db4c4f7039b8f426e4a997cedbe9ffdad1566786629ee7"}
Oct 29 23:45:44.241345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.241140    6203 scope.go:122] "RemoveContainer" containerID="819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a"
Oct 29 23:45:44.250578 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.250556    6203 scope.go:122] "RemoveContainer" containerID="819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a"
Oct 29 23:45:44.250934 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:44.250913    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a\": container with ID starting with 819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a not found: ID does not exist" containerID="819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a"
Oct 29 23:45:44.251005 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.250944    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a"} err="failed to get container status \"819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a\": rpc error: code = NotFound desc = could not find container \"819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a\": container with ID starting with 819b529d3c59a9203e64619bfc5129a049524005c58e991539a8fdfd17e2128a not found: ID does not exist"
Oct 29 23:45:44.298687 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.298653    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"]
Oct 29 23:45:44.303386 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:44.303356    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-06f91a/simulator-single-main-06f91a-kserve-555b666c95-zd2pf"]
Oct 29 23:45:46.017321 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:46.017285    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac392465-60bf-403d-a15b-80eb7087da46" path="/var/lib/kubelet/pods/ac392465-60bf-403d-a15b-80eb7087da46/volumes"
Oct 29 23:45:47.363945 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.363906    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"]
Oct 29 23:45:47.364562 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364539    6203 cpu_manager.go:414] "containerMap: removing container" podUID="59e2c046-6d89-4cf0-a340-94d81eb5eac6" containerName="revoke-keys"
Oct 29 23:45:47.364618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364565    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e2c046-6d89-4cf0-a340-94d81eb5eac6" containerName="revoke-keys"
Oct 29 23:45:47.364618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364607    6203 cpu_manager.go:414] "containerMap: removing container" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:47.364618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364615    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:47.364761 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364647    6203 cpu_manager.go:414] "containerMap: removing container" podUID="ac392465-60bf-403d-a15b-80eb7087da46" containerName="main"
Oct 29 23:45:47.364761 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364656    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac392465-60bf-403d-a15b-80eb7087da46" containerName="main"
Oct 29 23:45:47.364828 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364772    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e2c046-6d89-4cf0-a340-94d81eb5eac6" containerName="revoke-keys"
Oct 29 23:45:47.364828 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364792    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62da78b-e1d0-4067-adab-f80a06354e5e" containerName="cleanup"
Oct 29 23:45:47.364828 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.364804    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac392465-60bf-403d-a15b-80eb7087da46" containerName="main"
Oct 29 23:45:47.369225 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.369205    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.401632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.401603    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-04ae909c-kserve-self-signed-certs\""
Oct 29 23:45:47.403613 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.403585    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"]
Oct 29 23:45:47.487281 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.487242    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b1081-5047-45d4-ab2c-752fc28432a1-tls-certs\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.487455 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.487308    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-model-cache\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.487455 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.487382    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-dshm\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.487455 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.487436    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpk2\" (UniqueName: \"kubernetes.io/projected/692b1081-5047-45d4-ab2c-752fc28432a1-kube-api-access-hnpk2\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.487569 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.487526    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-tmp-dir\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.487569 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.487559    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-home\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589132 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589096    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpk2\" (UniqueName: \"kubernetes.io/projected/692b1081-5047-45d4-ab2c-752fc28432a1-kube-api-access-hnpk2\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589159    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-tmp-dir\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589185    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-home\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589216    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b1081-5047-45d4-ab2c-752fc28432a1-tls-certs\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589265    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-model-cache\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589337 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589314    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-dshm\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589679 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589655    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-tmp-dir\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589732 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589707    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-model-cache\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.589786 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.589770    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-home\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.591609 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.591587    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-dshm\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.592023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.592000    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b1081-5047-45d4-ab2c-752fc28432a1-tls-certs\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.612923 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.612890    6203 scope.go:122] "RemoveContainer" containerID="28f9e183e724141a23c2a48f86a2c111a21e67fee3b95aa31eede635cf50836b"
Oct 29 23:45:47.617012 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.616943    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpk2\" (UniqueName: \"kubernetes.io/projected/692b1081-5047-45d4-ab2c-752fc28432a1-kube-api-access-hnpk2\") pod \"e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") " pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.628908 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.628872    6203 scope.go:122] "RemoveContainer" containerID="cd25a2ee5efa07a0e70a32d3650e4d8ff49476096837334b4791ad79644a609a"
Oct 29 23:45:47.636934 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.636909    6203 scope.go:122] "RemoveContainer" containerID="6187036548c0df90a273efbb307465e3cdecfc380bffbbc9bf682b6521944eb4"
Oct 29 23:45:47.682514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.682487    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:47.983531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:47.983500    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"]
Oct 29 23:45:47.985767 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:45:47.985726    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod692b1081_5047_45d4_ab2c_752fc28432a1.slice/crio-79e4e66b845222298bd80a19e5702b13df5c5a9a2c504f6988852cd02a5a78ca WatchSource:0}: Error finding container 79e4e66b845222298bd80a19e5702b13df5c5a9a2c504f6988852cd02a5a78ca: Status 404 returned error can't find the container with id 79e4e66b845222298bd80a19e5702b13df5c5a9a2c504f6988852cd02a5a78ca
Oct 29 23:45:48.025562 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.025540    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:45:48.093596 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093554    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1361d43d-1046-4112-80da-c54ee34d5972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1361d43d-1046-4112-80da-c54ee34d5972-istio-podinfo\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.093762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093620    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-credential-socket\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.093762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093654    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1361d43d-1046-4112-80da-c54ee34d5972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1361d43d-1046-4112-80da-c54ee34d5972-istiod-ca-cert\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.093762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093687    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-kube-api-access-pxxnc\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-kube-api-access-pxxnc\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.093762 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093724    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-data\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.094068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093769    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-socket\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.094068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093798    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-istio-token\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-istio-token\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.094068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093909    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-envoy\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.094068 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.093958    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-certs\") pod \"1361d43d-1046-4112-80da-c54ee34d5972\" (UID: \"1361d43d-1046-4112-80da-c54ee34d5972\") "
Oct 29 23:45:48.094327 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.094303    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-socket" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:48.094537 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.094520    6203 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.094738 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.094712    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-certs" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:48.094895 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.094859    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1361d43d-1046-4112-80da-c54ee34d5972-istiod-ca-cert" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:48.094985 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.094964    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-credential-socket" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:48.095399 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.095372    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-data" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:48.096407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.096386    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-kube-api-access-pxxnc" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "kube-api-access-pxxnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:48.096644 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.096628    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1361d43d-1046-4112-80da-c54ee34d5972-istio-podinfo" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:45:48.097009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.096990    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-istio-token" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:48.097301 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.097275    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-envoy" pod "1361d43d-1046-4112-80da-c54ee34d5972" (UID: "1361d43d-1046-4112-80da-c54ee34d5972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:45:48.195818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195786    6203 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-envoy\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.195818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195813    6203 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-workload-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.195818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195823    6203 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1361d43d-1046-4112-80da-c54ee34d5972-istio-podinfo\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.196233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195831    6203 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-credential-socket\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.196233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195840    6203 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1361d43d-1046-4112-80da-c54ee34d5972-istiod-ca-cert\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.196233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195849    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxxnc\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-kube-api-access-pxxnc\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.196233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195857    6203 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1361d43d-1046-4112-80da-c54ee34d5972-istio-data\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.196233 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.195865    6203 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1361d43d-1046-4112-80da-c54ee34d5972-istio-token\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:48.271721 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.271680    6203 generic.go:356] "Generic (PLEG): container finished" podID="1361d43d-1046-4112-80da-c54ee34d5972" containerID="bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd" exitCode=0
Oct 29 23:45:48.271898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.271739    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"
Oct 29 23:45:48.271898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.271757    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" event={"ID":"1361d43d-1046-4112-80da-c54ee34d5972","Type":"ContainerDied","Data":"bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd"}
Oct 29 23:45:48.271898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.271806    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh" event={"ID":"1361d43d-1046-4112-80da-c54ee34d5972","Type":"ContainerDied","Data":"80453962f19627a5e9dbadaf73e2d63116f505425bb03096205b967c3a86057b"}
Oct 29 23:45:48.271898 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.271842    6203 scope.go:122] "RemoveContainer" containerID="bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd"
Oct 29 23:45:48.273449 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.273422    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz" event={"ID":"692b1081-5047-45d4-ab2c-752fc28432a1","Type":"ContainerStarted","Data":"61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20"}
Oct 29 23:45:48.273566 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.273451    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz" event={"ID":"692b1081-5047-45d4-ab2c-752fc28432a1","Type":"ContainerStarted","Data":"79e4e66b845222298bd80a19e5702b13df5c5a9a2c504f6988852cd02a5a78ca"}
Oct 29 23:45:48.281038 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.281022    6203 scope.go:122] "RemoveContainer" containerID="a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4"
Oct 29 23:45:48.289156 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.289139    6203 scope.go:122] "RemoveContainer" containerID="bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd"
Oct 29 23:45:48.289432 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:48.289415    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd\": container with ID starting with bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd not found: ID does not exist" containerID="bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd"
Oct 29 23:45:48.289472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.289440    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd"} err="failed to get container status \"bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd\": rpc error: code = NotFound desc = could not find container \"bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd\": container with ID starting with bae4595dc3d4f1b22ded32daa7675e7d6de776920547a6f6957c0d17d0f66afd not found: ID does not exist"
Oct 29 23:45:48.289472 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.289456    6203 scope.go:122] "RemoveContainer" containerID="a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4"
Oct 29 23:45:48.289698 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:45:48.289683    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4\": container with ID starting with a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4 not found: ID does not exist" containerID="a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4"
Oct 29 23:45:48.289744 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.289701    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4"} err="failed to get container status \"a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4\": rpc error: code = NotFound desc = could not find container \"a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4\": container with ID starting with a321e0abfd0d984efe55a771c9d3834925813003d8e1121cb42d74d22308ecd4 not found: ID does not exist"
Oct 29 23:45:48.335592 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.335538    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz" podStartSLOduration=1.3355249 podStartE2EDuration="1.3355249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:45:48.335268745 +0000 UTC m=+1926.439529131" watchObservedRunningTime="2026-10-29 23:45:48.3355249 +0000 UTC m=+1926.439785285"
Oct 29 23:45:48.376407 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.376373    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"]
Oct 29 23:45:48.380444 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.380424    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-06f91a-openshift-default-7ffc4567575grbh"]
Oct 29 23:45:48.682756 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:48.682723    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:49.806460 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:49.806420    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" podUID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:49.806912 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:49.806507    6203 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" podUID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:45:50.020161 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:50.020121    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1361d43d-1046-4112-80da-c54ee34d5972" path="/var/lib/kubelet/pods/1361d43d-1046-4112-80da-c54ee34d5972/volumes"
Oct 29 23:45:57.682969 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:57.682922    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:57.695336 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:57.695308    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:58.350015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:58.349984    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:45:59.533176 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.533149    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:45:59.560341 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.560317    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:45:59.560509 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.560491    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-plugins-config-volume\") pod \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") "
Oct 29 23:45:59.560581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.560530    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-plugins-config-volume\") pod \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") "
Oct 29 23:45:59.560640 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.560622    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-kube-api-access-pb6qb\" (UniqueName: \"kubernetes.io/projected/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-kube-api-access-pb6qb\") pod \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\" (UID: \"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60\") "
Oct 29 23:45:59.560693 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.560676    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-kube-api-access-8lzkt\" (UniqueName: \"kubernetes.io/projected/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-kube-api-access-8lzkt\") pod \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\" (UID: \"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9\") "
Oct 29 23:45:59.560925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.560899    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-plugins-config-volume" pod "5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" (UID: "5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:59.561001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.560920    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-plugins-config-volume" pod "5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" (UID: "5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:45:59.561059 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.561043    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:59.561102 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.561062    6203 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-plugins-config-volume\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:59.562599 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.562582    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-kube-api-access-8lzkt" pod "5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" (UID: "5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9"). InnerVolumeSpecName "kube-api-access-8lzkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:59.562645 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.562616    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-kube-api-access-pb6qb" pod "5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" (UID: "5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60"). InnerVolumeSpecName "kube-api-access-pb6qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:45:59.661773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.661737    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb6qb\" (UniqueName: \"kubernetes.io/projected/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60-kube-api-access-pb6qb\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:45:59.661773 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:45:59.661775    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzkt\" (UniqueName: \"kubernetes.io/projected/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9-kube-api-access-8lzkt\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:00.350426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.350340    6203 generic.go:356] "Generic (PLEG): container finished" podID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" containerID="81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621" exitCode=0
Oct 29 23:46:00.350426 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.350395    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"
Oct 29 23:46:00.350931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.350421    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" event={"ID":"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9","Type":"ContainerDied","Data":"81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621"}
Oct 29 23:46:00.350931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.350475    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb" event={"ID":"5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9","Type":"ContainerDied","Data":"69a0798fe989f1a7c396a7b3c59e4908bea6bdb45f75a1ca018fcd3208b8fdc3"}
Oct 29 23:46:00.350931 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.350506    6203 scope.go:122] "RemoveContainer" containerID="81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621"
Oct 29 23:46:00.352274 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.352252    6203 generic.go:356] "Generic (PLEG): container finished" podID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" containerID="16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081" exitCode=0
Oct 29 23:46:00.352396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.352305    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" event={"ID":"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60","Type":"ContainerDied","Data":"16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081"}
Oct 29 23:46:00.352396 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.352337    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c" event={"ID":"5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60","Type":"ContainerDied","Data":"fbdd353652f54da1e84ce5e5a9f623c1a3f8c8333ad7961d67e483fb9ebd397a"}
Oct 29 23:46:00.352524 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.352397    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"
Oct 29 23:46:00.359204 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.359186    6203 scope.go:122] "RemoveContainer" containerID="81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621"
Oct 29 23:46:00.359469 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:46:00.359451    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621\": container with ID starting with 81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621 not found: ID does not exist" containerID="81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621"
Oct 29 23:46:00.359525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.359478    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621"} err="failed to get container status \"81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621\": rpc error: code = NotFound desc = could not find container \"81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621\": container with ID starting with 81b9a5aebca52279665a39762ff5ae8c1f023eb9979bf6f31a508e5991e33621 not found: ID does not exist"
Oct 29 23:46:00.359525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.359494    6203 scope.go:122] "RemoveContainer" containerID="16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081"
Oct 29 23:46:00.367112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.367096    6203 scope.go:122] "RemoveContainer" containerID="16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081"
Oct 29 23:46:00.367371 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:46:00.367349    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081\": container with ID starting with 16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081 not found: ID does not exist" containerID="16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081"
Oct 29 23:46:00.367446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.367380    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081"} err="failed to get container status \"16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081\": rpc error: code = NotFound desc = could not find container \"16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081\": container with ID starting with 16b7e1a4b0be1d888440a0cba5341e0308a2cb56f5f198a442fee58d272f6081 not found: ID does not exist"
Oct 29 23:46:00.410084 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.410058    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"]
Oct 29 23:46:00.424618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.424590    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-06f91a-548985plsb"]
Oct 29 23:46:00.452618 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.452592    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"]
Oct 29 23:46:00.458534 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:00.458509    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-06f91a-6568485f8gvj6c"]
Oct 29 23:46:02.017505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:02.017469    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" path="/var/lib/kubelet/pods/5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9/volumes"
Oct 29 23:46:02.018071 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:02.018038    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" path="/var/lib/kubelet/pods/5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60/volumes"
Oct 29 23:46:03.452381 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452344    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"]
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452720    6203 cpu_manager.go:414] "containerMap: removing container" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452732    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452743    6203 cpu_manager.go:414] "containerMap: removing container" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452747    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452758    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" containerName="payload-pre-processing"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452763    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" containerName="payload-pre-processing"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452788    6203 cpu_manager.go:414] "containerMap: removing container" podUID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" containerName="payload-processing"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452793    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" containerName="payload-processing"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452859    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fce1fa9-8e0e-4d3c-a9aa-84c3e2f57d60" containerName="payload-processing"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452870    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452890    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="1361d43d-1046-4112-80da-c54ee34d5972" containerName="istio-proxy"
Oct 29 23:46:03.452958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.452896    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6f1e95-3d0d-42d9-84b5-cc169c1e06e9" containerName="payload-pre-processing"
Oct 29 23:46:03.457802 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.457778    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.490267 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.490233    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-04ae909c-kserve-self-signed-certs\""
Oct 29 23:46:03.491862 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.491842    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"]
Oct 29 23:46:03.497354 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.497332    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6833cdae-74a8-42f1-9c29-9fc741cd4020-tls-certs\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.497493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.497383    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-tmp-dir\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.497493 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.497420    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-home\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.497632 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.497603    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwvq\" (UniqueName: \"kubernetes.io/projected/6833cdae-74a8-42f1-9c29-9fc741cd4020-kube-api-access-nrwvq\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.497682 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.497666    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-dshm\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.497721 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.497701    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-model-cache\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599025    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-home\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599262 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599074    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwvq\" (UniqueName: \"kubernetes.io/projected/6833cdae-74a8-42f1-9c29-9fc741cd4020-kube-api-access-nrwvq\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599262 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599193    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-dshm\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599262 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599230    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-model-cache\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599276    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6833cdae-74a8-42f1-9c29-9fc741cd4020-tls-certs\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599438 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599331    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-tmp-dir\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599443    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-home\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599645 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599629    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-tmp-dir\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.599696 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.599640    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-model-cache\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.601432 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.601404    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-dshm\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.601714 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.601696    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6833cdae-74a8-42f1-9c29-9fc741cd4020-tls-certs\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.624893 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.624860    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwvq\" (UniqueName: \"kubernetes.io/projected/6833cdae-74a8-42f1-9c29-9fc741cd4020-kube-api-access-nrwvq\") pod \"e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") " pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:03.772852 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:03.772762    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:04.073222 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:04.073194    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"]
Oct 29 23:46:04.073666 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:46:04.073640    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6833cdae_74a8_42f1_9c29_9fc741cd4020.slice/crio-3ba0ccd241a7fa13f3d8770c03fd02325f2c60b22dbe20beb305f3f875aa7490 WatchSource:0}: Error finding container 3ba0ccd241a7fa13f3d8770c03fd02325f2c60b22dbe20beb305f3f875aa7490: Status 404 returned error can't find the container with id 3ba0ccd241a7fa13f3d8770c03fd02325f2c60b22dbe20beb305f3f875aa7490
Oct 29 23:46:04.375015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:04.374919    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w" event={"ID":"6833cdae-74a8-42f1-9c29-9fc741cd4020","Type":"ContainerStarted","Data":"efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7"}
Oct 29 23:46:04.375015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:04.374964    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w" event={"ID":"6833cdae-74a8-42f1-9c29-9fc741cd4020","Type":"ContainerStarted","Data":"3ba0ccd241a7fa13f3d8770c03fd02325f2c60b22dbe20beb305f3f875aa7490"}
Oct 29 23:46:04.773812 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:04.773757    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:09.923715 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:46:09.923680    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:46:13.773342 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:13.773308    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:13.786198 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:13.786168    6203 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:13.832887 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:13.832817    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w" podStartSLOduration=10.832804467 podStartE2EDuration="10.832804467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:04.420606263 +0000 UTC m=+1942.524866645" watchObservedRunningTime="2026-10-29 23:46:13.832804467 +0000 UTC m=+1951.937064853"
Oct 29 23:46:14.446998 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:14.446966    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:25.678127 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.678090    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"]
Oct 29 23:46:25.681768 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.681749    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.700819 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.700792    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:46:25.715424 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.715395    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:46:25.715567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.715395    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:46:25.715567 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.715458    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:46:25.717250 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.717229    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"]
Oct 29 23:46:25.770595 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.770559    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh976\" (UniqueName: \"kubernetes.io/projected/210fbfef-0dbf-4d98-a293-7aac91094004-kube-api-access-bh976\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.770775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.770607    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.770775 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.770726    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.871922 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.871874    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh976\" (UniqueName: \"kubernetes.io/projected/210fbfef-0dbf-4d98-a293-7aac91094004-kube-api-access-bh976\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.872108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.872019    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.872108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.872064    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.872712 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.872686    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.874477 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.874456    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.898293 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.898268    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh976\" (UniqueName: \"kubernetes.io/projected/210fbfef-0dbf-4d98-a293-7aac91094004-kube-api-access-bh976\") pod \"maas-api-revoke-sub-452b6d7b-hzdkl\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:25.995138 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:25.995047    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:26.295048 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:26.295018    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"]
Oct 29 23:46:26.296548 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:46:26.296517    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210fbfef_0dbf_4d98_a293_7aac91094004.slice/crio-036a5b1d87b898aa04aef8dbeff42cfc8bff4ffaf7c533f23508dcc76bedf891 WatchSource:0}: Error finding container 036a5b1d87b898aa04aef8dbeff42cfc8bff4ffaf7c533f23508dcc76bedf891: Status 404 returned error can't find the container with id 036a5b1d87b898aa04aef8dbeff42cfc8bff4ffaf7c533f23508dcc76bedf891
Oct 29 23:46:26.496684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:26.496648    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl" event={"ID":"210fbfef-0dbf-4d98-a293-7aac91094004","Type":"ContainerStarted","Data":"036a5b1d87b898aa04aef8dbeff42cfc8bff4ffaf7c533f23508dcc76bedf891"}
Oct 29 23:46:27.504932 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:27.504873    6203 generic.go:356] "Generic (PLEG): container finished" podID="210fbfef-0dbf-4d98-a293-7aac91094004" containerID="5ec612b1444b3411b1db007e9cd5d264830279185b432a66174c99b65bd19eec" exitCode=0
Oct 29 23:46:27.505476 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:27.504955    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl" event={"ID":"210fbfef-0dbf-4d98-a293-7aac91094004","Type":"ContainerDied","Data":"5ec612b1444b3411b1db007e9cd5d264830279185b432a66174c99b65bd19eec"}
Oct 29 23:46:28.930581 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:28.930282    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:29.020851 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.020811    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-xks-ca\") pod \"210fbfef-0dbf-4d98-a293-7aac91094004\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") "
Oct 29 23:46:29.020851 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.020857    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-openshift-ca\") pod \"210fbfef-0dbf-4d98-a293-7aac91094004\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") "
Oct 29 23:46:29.021133 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.020917    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210fbfef-0dbf-4d98-a293-7aac91094004-kube-api-access-bh976\" (UniqueName: \"kubernetes.io/projected/210fbfef-0dbf-4d98-a293-7aac91094004-kube-api-access-bh976\") pod \"210fbfef-0dbf-4d98-a293-7aac91094004\" (UID: \"210fbfef-0dbf-4d98-a293-7aac91094004\") "
Oct 29 23:46:29.021309 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.021286    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-openshift-ca" pod "210fbfef-0dbf-4d98-a293-7aac91094004" (UID: "210fbfef-0dbf-4d98-a293-7aac91094004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:29.023023 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.022995    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-xks-ca" pod "210fbfef-0dbf-4d98-a293-7aac91094004" (UID: "210fbfef-0dbf-4d98-a293-7aac91094004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:29.023130 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.023042    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210fbfef-0dbf-4d98-a293-7aac91094004-kube-api-access-bh976" pod "210fbfef-0dbf-4d98-a293-7aac91094004" (UID: "210fbfef-0dbf-4d98-a293-7aac91094004"). InnerVolumeSpecName "kube-api-access-bh976". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:29.122488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.122450    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:29.122488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.122483    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/210fbfef-0dbf-4d98-a293-7aac91094004-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:29.122488 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.122493    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh976\" (UniqueName: \"kubernetes.io/projected/210fbfef-0dbf-4d98-a293-7aac91094004-kube-api-access-bh976\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:29.523827 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.523799    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl"
Oct 29 23:46:29.523827 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.523803    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hzdkl" event={"ID":"210fbfef-0dbf-4d98-a293-7aac91094004","Type":"ContainerDied","Data":"036a5b1d87b898aa04aef8dbeff42cfc8bff4ffaf7c533f23508dcc76bedf891"}
Oct 29 23:46:29.524081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:29.523905    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="036a5b1d87b898aa04aef8dbeff42cfc8bff4ffaf7c533f23508dcc76bedf891"
Oct 29 23:46:37.063542 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:37.063506    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"]
Oct 29 23:46:39.929650 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:46:39.929603    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:46:43.086117 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:43.086084    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"]
Oct 29 23:46:47.673979 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.673939    6203 scope.go:122] "RemoveContainer" containerID="a07a0695cfeecb8fad1798ad3bfe136f4f6ef843090b6efc586f2988b2d65b45"
Oct 29 23:46:47.683550 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.683510    6203 scope.go:122] "RemoveContainer" containerID="b6562b0a077f22112cea0dd8b43c6ab5f274d5ea1a473eb3469b3190ff54fb1b"
Oct 29 23:46:47.684741 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.684719    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"]
Oct 29 23:46:47.685206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.685188    6203 cpu_manager.go:414] "containerMap: removing container" podUID="210fbfef-0dbf-4d98-a293-7aac91094004" containerName="revoke-keys"
Oct 29 23:46:47.685206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.685208    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210fbfef-0dbf-4d98-a293-7aac91094004" containerName="revoke-keys"
Oct 29 23:46:47.685333 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.685308    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="210fbfef-0dbf-4d98-a293-7aac91094004" containerName="revoke-keys"
Oct 29 23:46:47.689921 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.689843    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.693511 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.693494    6203 scope.go:122] "RemoveContainer" containerID="19a15c631e863e17c7f8d0f48b8b98ce9dfce588e923d90730808dd966b87a04"
Oct 29 23:46:47.698980 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.698947    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.699088 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.698981    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdks6\" (UniqueName: \"kubernetes.io/projected/6a81917f-558d-4676-8767-67b502e6a4e2-kube-api-access-bdks6\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.699151 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.699099    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.709205 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.709179    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:46:47.709345 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.709213    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:46:47.724193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.723974    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:46:47.724312 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.724294    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:46:47.726089 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.726064    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"]
Oct 29 23:46:47.800036 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.800001    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.800206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.800042    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdks6\" (UniqueName: \"kubernetes.io/projected/6a81917f-558d-4676-8767-67b502e6a4e2-kube-api-access-bdks6\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.800206 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.800169    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.800759 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.800739    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.802342 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.802315    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:47.825804 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:47.825775    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdks6\" (UniqueName: \"kubernetes.io/projected/6a81917f-558d-4676-8767-67b502e6a4e2-kube-api-access-bdks6\") pod \"maas-api-revoke-sub-3e31a506-p9jwr\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:48.010446 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:48.010316    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:48.324701 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:48.324664    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"]
Oct 29 23:46:48.326634 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:46:48.326602    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a81917f_558d_4676_8767_67b502e6a4e2.slice/crio-764a00c157cd3d2bc1112ee0e08e39acb6836649429bfa0563e636b079c63eba WatchSource:0}: Error finding container 764a00c157cd3d2bc1112ee0e08e39acb6836649429bfa0563e636b079c63eba: Status 404 returned error can't find the container with id 764a00c157cd3d2bc1112ee0e08e39acb6836649429bfa0563e636b079c63eba
Oct 29 23:46:48.676939 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:48.676861    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr" event={"ID":"6a81917f-558d-4676-8767-67b502e6a4e2","Type":"ContainerStarted","Data":"b4fb6a8a504d15873c17104d6848298625c76e57be5ba580d4badf3e9116dffd"}
Oct 29 23:46:48.677484 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:48.677007    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr" event={"ID":"6a81917f-558d-4676-8767-67b502e6a4e2","Type":"ContainerStarted","Data":"764a00c157cd3d2bc1112ee0e08e39acb6836649429bfa0563e636b079c63eba"}
Oct 29 23:46:49.037006 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:49.033007    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr" podStartSLOduration=2.032985687 podStartE2EDuration="2.032985687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:46:48.739392024 +0000 UTC m=+1986.843652412" watchObservedRunningTime="2026-10-29 23:46:49.032985687 +0000 UTC m=+1987.137246074"
Oct 29 23:46:49.050474 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:49.037956    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"]
Oct 29 23:46:49.050474 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:49.041269    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33257fa5-knntm"]
Oct 29 23:46:49.696901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:49.692964    6203 generic.go:356] "Generic (PLEG): container finished" podID="6a81917f-558d-4676-8767-67b502e6a4e2" containerID="b4fb6a8a504d15873c17104d6848298625c76e57be5ba580d4badf3e9116dffd" exitCode=0
Oct 29 23:46:49.696901 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:49.693086    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr" event={"ID":"6a81917f-558d-4676-8767-67b502e6a4e2","Type":"ContainerDied","Data":"b4fb6a8a504d15873c17104d6848298625c76e57be5ba580d4badf3e9116dffd"}
Oct 29 23:46:50.019201 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.019121    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0792e31-57fc-4172-9956-e8c1a289f739" path="/var/lib/kubelet/pods/d0792e31-57fc-4172-9956-e8c1a289f739/volumes"
Oct 29 23:46:50.826844 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.826819    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:50.845049 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.845017    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-openshift-ca\") pod \"6a81917f-558d-4676-8767-67b502e6a4e2\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") "
Oct 29 23:46:50.845214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.845073    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-xks-ca\") pod \"6a81917f-558d-4676-8767-67b502e6a4e2\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") "
Oct 29 23:46:50.845214 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.845184    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a81917f-558d-4676-8767-67b502e6a4e2-kube-api-access-bdks6\" (UniqueName: \"kubernetes.io/projected/6a81917f-558d-4676-8767-67b502e6a4e2-kube-api-access-bdks6\") pod \"6a81917f-558d-4676-8767-67b502e6a4e2\" (UID: \"6a81917f-558d-4676-8767-67b502e6a4e2\") "
Oct 29 23:46:50.845412 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.845378    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-openshift-ca" pod "6a81917f-558d-4676-8767-67b502e6a4e2" (UID: "6a81917f-558d-4676-8767-67b502e6a4e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:46:50.845587 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.845567    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:50.847252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.847221    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-xks-ca" pod "6a81917f-558d-4676-8767-67b502e6a4e2" (UID: "6a81917f-558d-4676-8767-67b502e6a4e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:50.847252 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.847232    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a81917f-558d-4676-8767-67b502e6a4e2-kube-api-access-bdks6" pod "6a81917f-558d-4676-8767-67b502e6a4e2" (UID: "6a81917f-558d-4676-8767-67b502e6a4e2"). InnerVolumeSpecName "kube-api-access-bdks6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:50.946709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.946677    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdks6\" (UniqueName: \"kubernetes.io/projected/6a81917f-558d-4676-8767-67b502e6a4e2-kube-api-access-bdks6\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:50.946709 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:50.946705    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a81917f-558d-4676-8767-67b502e6a4e2-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:51.704035 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:51.704000    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr"
Oct 29 23:46:51.704035 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:51.704025    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-p9jwr" event={"ID":"6a81917f-558d-4676-8767-67b502e6a4e2","Type":"ContainerDied","Data":"764a00c157cd3d2bc1112ee0e08e39acb6836649429bfa0563e636b079c63eba"}
Oct 29 23:46:51.704271 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:51.704078    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764a00c157cd3d2bc1112ee0e08e39acb6836649429bfa0563e636b079c63eba"
Oct 29 23:46:52.083119 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.083030    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz" podUID="692b1081-5047-45d4-ab2c-752fc28432a1" containerName="main" containerID="cri-o://61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20" gracePeriod=45
Oct 29 23:46:52.329128 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.329099    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:46:52.360557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360483    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-model-cache\") pod \"692b1081-5047-45d4-ab2c-752fc28432a1\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") "
Oct 29 23:46:52.360557 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360533    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692b1081-5047-45d4-ab2c-752fc28432a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/692b1081-5047-45d4-ab2c-752fc28432a1-tls-certs\") pod \"692b1081-5047-45d4-ab2c-752fc28432a1\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") "
Oct 29 23:46:52.360764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360562    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-home\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-home\") pod \"692b1081-5047-45d4-ab2c-752fc28432a1\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") "
Oct 29 23:46:52.360764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360637    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692b1081-5047-45d4-ab2c-752fc28432a1-kube-api-access-hnpk2\" (UniqueName: \"kubernetes.io/projected/692b1081-5047-45d4-ab2c-752fc28432a1-kube-api-access-hnpk2\") pod \"692b1081-5047-45d4-ab2c-752fc28432a1\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") "
Oct 29 23:46:52.360764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360660    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-tmp-dir\") pod \"692b1081-5047-45d4-ab2c-752fc28432a1\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") "
Oct 29 23:46:52.360764 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360742    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-dshm\") pod \"692b1081-5047-45d4-ab2c-752fc28432a1\" (UID: \"692b1081-5047-45d4-ab2c-752fc28432a1\") "
Oct 29 23:46:52.361027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360754    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-model-cache" pod "692b1081-5047-45d4-ab2c-752fc28432a1" (UID: "692b1081-5047-45d4-ab2c-752fc28432a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:52.361027 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.360899    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-home" pod "692b1081-5047-45d4-ab2c-752fc28432a1" (UID: "692b1081-5047-45d4-ab2c-752fc28432a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:52.361120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.361036    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-tmp-dir" pod "692b1081-5047-45d4-ab2c-752fc28432a1" (UID: "692b1081-5047-45d4-ab2c-752fc28432a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:52.361120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.361060    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:52.361120 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.361078    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:52.362837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.362807    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692b1081-5047-45d4-ab2c-752fc28432a1-tls-certs" pod "692b1081-5047-45d4-ab2c-752fc28432a1" (UID: "692b1081-5047-45d4-ab2c-752fc28432a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:52.362837 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.362814    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-dshm" pod "692b1081-5047-45d4-ab2c-752fc28432a1" (UID: "692b1081-5047-45d4-ab2c-752fc28432a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:52.363042 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.362846    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692b1081-5047-45d4-ab2c-752fc28432a1-kube-api-access-hnpk2" pod "692b1081-5047-45d4-ab2c-752fc28432a1" (UID: "692b1081-5047-45d4-ab2c-752fc28432a1"). InnerVolumeSpecName "kube-api-access-hnpk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:52.461629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.461596    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpk2\" (UniqueName: \"kubernetes.io/projected/692b1081-5047-45d4-ab2c-752fc28432a1-kube-api-access-hnpk2\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:52.461629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.461624    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:52.461629 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.461635    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b1081-5047-45d4-ab2c-752fc28432a1-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:52.461897 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.461644    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b1081-5047-45d4-ab2c-752fc28432a1-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:52.710196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.710154    6203 generic.go:356] "Generic (PLEG): container finished" podID="692b1081-5047-45d4-ab2c-752fc28432a1" containerID="61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20" exitCode=0
Oct 29 23:46:52.710196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.710197    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"
Oct 29 23:46:52.710456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.710246    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz" event={"ID":"692b1081-5047-45d4-ab2c-752fc28432a1","Type":"ContainerDied","Data":"61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20"}
Oct 29 23:46:52.710456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.710291    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz" event={"ID":"692b1081-5047-45d4-ab2c-752fc28432a1","Type":"ContainerDied","Data":"79e4e66b845222298bd80a19e5702b13df5c5a9a2c504f6988852cd02a5a78ca"}
Oct 29 23:46:52.710456 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.710312    6203 scope.go:122] "RemoveContainer" containerID="61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20"
Oct 29 23:46:52.719570 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.719552    6203 scope.go:122] "RemoveContainer" containerID="61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20"
Oct 29 23:46:52.719838 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:46:52.719811    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20\": container with ID starting with 61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20 not found: ID does not exist" containerID="61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20"
Oct 29 23:46:52.719959 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.719844    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20"} err="failed to get container status \"61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20\": rpc error: code = NotFound desc = could not find container \"61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20\": container with ID starting with 61baa270f5cd20b1f1cf7a6e4be8dd7101484768ac2516a5952d1665e84b0c20 not found: ID does not exist"
Oct 29 23:46:52.769647 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.769616    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"]
Oct 29 23:46:52.784001 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:52.783973    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-04ae909c-kserve-7fcfb94c57-npsgz"]
Oct 29 23:46:54.017237 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:54.017160    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692b1081-5047-45d4-ab2c-752fc28432a1" path="/var/lib/kubelet/pods/692b1081-5047-45d4-ab2c-752fc28432a1/volumes"
Oct 29 23:46:58.106234 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.106194    6203 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w" podUID="6833cdae-74a8-42f1-9c29-9fc741cd4020" containerName="main" containerID="cri-o://efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7" gracePeriod=45
Oct 29 23:46:58.368166 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.368111    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:58.416562 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.416530    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6833cdae-74a8-42f1-9c29-9fc741cd4020-kube-api-access-nrwvq\" (UniqueName: \"kubernetes.io/projected/6833cdae-74a8-42f1-9c29-9fc741cd4020-kube-api-access-nrwvq\") pod \"6833cdae-74a8-42f1-9c29-9fc741cd4020\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") "
Oct 29 23:46:58.416767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.416582    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-home\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-home\") pod \"6833cdae-74a8-42f1-9c29-9fc741cd4020\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") "
Oct 29 23:46:58.416767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.416628    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-tmp-dir\") pod \"6833cdae-74a8-42f1-9c29-9fc741cd4020\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") "
Oct 29 23:46:58.416767 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.416688    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-model-cache\") pod \"6833cdae-74a8-42f1-9c29-9fc741cd4020\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") "
Oct 29 23:46:58.416992 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.416777    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-dshm\") pod \"6833cdae-74a8-42f1-9c29-9fc741cd4020\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") "
Oct 29 23:46:58.416992 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.416853    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6833cdae-74a8-42f1-9c29-9fc741cd4020-tls-certs\" (UniqueName: \"kubernetes.io/secret/6833cdae-74a8-42f1-9c29-9fc741cd4020-tls-certs\") pod \"6833cdae-74a8-42f1-9c29-9fc741cd4020\" (UID: \"6833cdae-74a8-42f1-9c29-9fc741cd4020\") "
Oct 29 23:46:58.417190 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.417164    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-tmp-dir" pod "6833cdae-74a8-42f1-9c29-9fc741cd4020" (UID: "6833cdae-74a8-42f1-9c29-9fc741cd4020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:58.417261 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.417195    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-home" pod "6833cdae-74a8-42f1-9c29-9fc741cd4020" (UID: "6833cdae-74a8-42f1-9c29-9fc741cd4020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:58.417486 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.417453    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-model-cache" pod "6833cdae-74a8-42f1-9c29-9fc741cd4020" (UID: "6833cdae-74a8-42f1-9c29-9fc741cd4020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:58.419428 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.419400    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6833cdae-74a8-42f1-9c29-9fc741cd4020-tls-certs" pod "6833cdae-74a8-42f1-9c29-9fc741cd4020" (UID: "6833cdae-74a8-42f1-9c29-9fc741cd4020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:46:58.419642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.419625    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6833cdae-74a8-42f1-9c29-9fc741cd4020-kube-api-access-nrwvq" pod "6833cdae-74a8-42f1-9c29-9fc741cd4020" (UID: "6833cdae-74a8-42f1-9c29-9fc741cd4020"). InnerVolumeSpecName "kube-api-access-nrwvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:46:58.419689 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.419671    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-dshm" pod "6833cdae-74a8-42f1-9c29-9fc741cd4020" (UID: "6833cdae-74a8-42f1-9c29-9fc741cd4020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:46:58.517809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.517773    6203 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-dshm\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:58.517809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.517803    6203 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6833cdae-74a8-42f1-9c29-9fc741cd4020-tls-certs\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:58.517809 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.517814    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrwvq\" (UniqueName: \"kubernetes.io/projected/6833cdae-74a8-42f1-9c29-9fc741cd4020-kube-api-access-nrwvq\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:58.518108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.517823    6203 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-home\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:58.518108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.517833    6203 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-tmp-dir\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:58.518108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.517843    6203 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6833cdae-74a8-42f1-9c29-9fc741cd4020-model-cache\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:46:58.741818 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.741781    6203 generic.go:356] "Generic (PLEG): container finished" podID="6833cdae-74a8-42f1-9c29-9fc741cd4020" containerID="efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7" exitCode=0
Oct 29 23:46:58.742031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.741827    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"
Oct 29 23:46:58.742031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.741862    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w" event={"ID":"6833cdae-74a8-42f1-9c29-9fc741cd4020","Type":"ContainerDied","Data":"efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7"}
Oct 29 23:46:58.742031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.741919    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w" event={"ID":"6833cdae-74a8-42f1-9c29-9fc741cd4020","Type":"ContainerDied","Data":"3ba0ccd241a7fa13f3d8770c03fd02325f2c60b22dbe20beb305f3f875aa7490"}
Oct 29 23:46:58.742031 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.741937    6203 scope.go:122] "RemoveContainer" containerID="efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7"
Oct 29 23:46:58.751344 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.751326    6203 scope.go:122] "RemoveContainer" containerID="efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7"
Oct 29 23:46:58.751628 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:46:58.751604    6203 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7\": container with ID starting with efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7 not found: ID does not exist" containerID="efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7"
Oct 29 23:46:58.751671 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.751638    6203 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7"} err="failed to get container status \"efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7\": rpc error: code = NotFound desc = could not find container \"efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7\": container with ID starting with efeaf6fc573497d9fb01882d0feabbada8635cebab2c1c26c00fa916a91a7fb7 not found: ID does not exist"
Oct 29 23:46:58.800576 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.800542    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"]
Oct 29 23:46:58.805253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:46:58.805218    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-04ae909c-kserve-66b9947cc7-mmt6w"]
Oct 29 23:47:00.017346 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:00.017311    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6833cdae-74a8-42f1-9c29-9fc741cd4020" path="/var/lib/kubelet/pods/6833cdae-74a8-42f1-9c29-9fc741cd4020/volumes"
Oct 29 23:47:07.053291 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.053247    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"]
Oct 29 23:47:07.053857 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.053836    6203 cpu_manager.go:414] "containerMap: removing container" podUID="6833cdae-74a8-42f1-9c29-9fc741cd4020" containerName="main"
Oct 29 23:47:07.053925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.053860    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6833cdae-74a8-42f1-9c29-9fc741cd4020" containerName="main"
Oct 29 23:47:07.053925 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.053916    6203 cpu_manager.go:414] "containerMap: removing container" podUID="6a81917f-558d-4676-8767-67b502e6a4e2" containerName="revoke-keys"
Oct 29 23:47:07.054007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.053925    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a81917f-558d-4676-8767-67b502e6a4e2" containerName="revoke-keys"
Oct 29 23:47:07.054007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.053966    6203 cpu_manager.go:414] "containerMap: removing container" podUID="692b1081-5047-45d4-ab2c-752fc28432a1" containerName="main"
Oct 29 23:47:07.054007 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.053975    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692b1081-5047-45d4-ab2c-752fc28432a1" containerName="main"
Oct 29 23:47:07.054099 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.054074    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="692b1081-5047-45d4-ab2c-752fc28432a1" containerName="main"
Oct 29 23:47:07.054099 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.054092    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="6833cdae-74a8-42f1-9c29-9fc741cd4020" containerName="main"
Oct 29 23:47:07.054155 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.054106    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a81917f-558d-4676-8767-67b502e6a4e2" containerName="revoke-keys"
Oct 29 23:47:07.057658 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.057633    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.076815 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.076780    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"]
Oct 29 23:47:07.085064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.085035    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:47:07.085064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.085066    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:47:07.088660 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.088637    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fp9sw\""
Oct 29 23:47:07.088792 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.088676    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:47:07.124944 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.124913    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.125081 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.124974    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.125139 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.125111    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wlqb\" (UniqueName: \"kubernetes.io/projected/a9458015-0a10-421f-bade-7fbb1c19893b-kube-api-access-9wlqb\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.226409 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.226371    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.226619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.226438    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.226619 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.226481    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wlqb\" (UniqueName: \"kubernetes.io/projected/a9458015-0a10-421f-bade-7fbb1c19893b-kube-api-access-9wlqb\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.227188 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.227163    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.228799 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.228777    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.252324 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.252295    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wlqb\" (UniqueName: \"kubernetes.io/projected/a9458015-0a10-421f-bade-7fbb1c19893b-kube-api-access-9wlqb\") pod \"maas-api-revoke-sub-d74bef3b-k4h4v\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.370480 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.370389    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:07.668735 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.668709    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"]
Oct 29 23:47:07.670420 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:47:07.670389    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9458015_0a10_421f_bade_7fbb1c19893b.slice/crio-3a625963c1f3028eddbfe77ba5896c7c0294b58c2df5f131b66128ec41cb6f57 WatchSource:0}: Error finding container 3a625963c1f3028eddbfe77ba5896c7c0294b58c2df5f131b66128ec41cb6f57: Status 404 returned error can't find the container with id 3a625963c1f3028eddbfe77ba5896c7c0294b58c2df5f131b66128ec41cb6f57
Oct 29 23:47:07.807933 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:07.807896    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v" event={"ID":"a9458015-0a10-421f-bade-7fbb1c19893b","Type":"ContainerStarted","Data":"3a625963c1f3028eddbfe77ba5896c7c0294b58c2df5f131b66128ec41cb6f57"}
Oct 29 23:47:08.815242 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:08.815157    6203 generic.go:356] "Generic (PLEG): container finished" podID="a9458015-0a10-421f-bade-7fbb1c19893b" containerID="029828def325ca0f3f37f05414a48e256adfcdd902a4a5a67e67751926c96d43" exitCode=0
Oct 29 23:47:08.815616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:08.815242    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v" event={"ID":"a9458015-0a10-421f-bade-7fbb1c19893b","Type":"ContainerDied","Data":"029828def325ca0f3f37f05414a48e256adfcdd902a4a5a67e67751926c96d43"}
Oct 29 23:47:09.925425 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:47:09.925393    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:47:09.949541 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:09.949516    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:10.053685 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.053656    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9458015-0a10-421f-bade-7fbb1c19893b-kube-api-access-9wlqb\" (UniqueName: \"kubernetes.io/projected/a9458015-0a10-421f-bade-7fbb1c19893b-kube-api-access-9wlqb\") pod \"a9458015-0a10-421f-bade-7fbb1c19893b\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") "
Oct 29 23:47:10.053859 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.053691    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-openshift-ca\") pod \"a9458015-0a10-421f-bade-7fbb1c19893b\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") "
Oct 29 23:47:10.053859 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.053808    6203 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-xks-ca\") pod \"a9458015-0a10-421f-bade-7fbb1c19893b\" (UID: \"a9458015-0a10-421f-bade-7fbb1c19893b\") "
Oct 29 23:47:10.054197 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.054156    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-openshift-ca" pod "a9458015-0a10-421f-bade-7fbb1c19893b" (UID: "a9458015-0a10-421f-bade-7fbb1c19893b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:47:10.055900 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.055861    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9458015-0a10-421f-bade-7fbb1c19893b-kube-api-access-9wlqb" pod "a9458015-0a10-421f-bade-7fbb1c19893b" (UID: "a9458015-0a10-421f-bade-7fbb1c19893b"). InnerVolumeSpecName "kube-api-access-9wlqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:47:10.056037 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.056017    6203 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-xks-ca" pod "a9458015-0a10-421f-bade-7fbb1c19893b" (UID: "a9458015-0a10-421f-bade-7fbb1c19893b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:47:10.154888 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.154800    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-xks-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:47:10.154888 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.154827    6203 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wlqb\" (UniqueName: \"kubernetes.io/projected/a9458015-0a10-421f-bade-7fbb1c19893b-kube-api-access-9wlqb\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:47:10.154888 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.154837    6203 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9458015-0a10-421f-bade-7fbb1c19893b-maas-api-openshift-ca\") on node \"ip-10-0-139-224.ec2.internal\" DevicePath \"\""
Oct 29 23:47:10.827385 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.827347    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v" event={"ID":"a9458015-0a10-421f-bade-7fbb1c19893b","Type":"ContainerDied","Data":"3a625963c1f3028eddbfe77ba5896c7c0294b58c2df5f131b66128ec41cb6f57"}
Oct 29 23:47:10.827648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.827397    6203 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a625963c1f3028eddbfe77ba5896c7c0294b58c2df5f131b66128ec41cb6f57"
Oct 29 23:47:10.827648 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:10.827364    6203 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74bef3b-k4h4v"
Oct 29 23:47:22.028505 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:22.028469    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"]
Oct 29 23:47:22.043009 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:22.042980    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gk5pf"]
Oct 29 23:47:24.021513 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:24.021479    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746c044c-105d-4d84-bb5d-e16ba4c3cdce" path="/var/lib/kubelet/pods/746c044c-105d-4d84-bb5d-e16ba4c3cdce/volumes"
Oct 29 23:47:27.491630 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.491588    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-n96br"]
Oct 29 23:47:27.492265 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.492222    6203 cpu_manager.go:414] "containerMap: removing container" podUID="a9458015-0a10-421f-bade-7fbb1c19893b" containerName="revoke-keys"
Oct 29 23:47:27.492265 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.492244    6203 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9458015-0a10-421f-bade-7fbb1c19893b" containerName="revoke-keys"
Oct 29 23:47:27.492404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.492353    6203 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9458015-0a10-421f-bade-7fbb1c19893b" containerName="revoke-keys"
Oct 29 23:47:27.496867 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.496844    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.516261 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.516229    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:47:27.530993 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.530953    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:47:27.531108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.531010    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:47:27.531108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.531034    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k7tlz\""
Oct 29 23:47:27.531108 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.531059    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 23:47:27.533494 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.533471    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-n96br"]
Oct 29 23:47:27.626203 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.626167    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rfj\" (UniqueName: \"kubernetes.io/projected/25073edb-2065-4a24-8ef1-3a6ed6c46b28-kube-api-access-g8rfj\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.626392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.626227    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073edb-2065-4a24-8ef1-3a6ed6c46b28-maas-api-tls\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.626392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.626328    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073edb-2065-4a24-8ef1-3a6ed6c46b28-metrics-certs\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.626392 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.626383    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073edb-2065-4a24-8ef1-3a6ed6c46b28-openshift-service-ca\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.727129 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.727097    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073edb-2065-4a24-8ef1-3a6ed6c46b28-maas-api-tls\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.727316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.727146    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073edb-2065-4a24-8ef1-3a6ed6c46b28-metrics-certs\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.727316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.727189    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073edb-2065-4a24-8ef1-3a6ed6c46b28-openshift-service-ca\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.727316 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.727227    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rfj\" (UniqueName: \"kubernetes.io/projected/25073edb-2065-4a24-8ef1-3a6ed6c46b28-kube-api-access-g8rfj\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.727928 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.727903    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25073edb-2065-4a24-8ef1-3a6ed6c46b28-openshift-service-ca\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.730178 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.730154    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25073edb-2065-4a24-8ef1-3a6ed6c46b28-maas-api-tls\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.730317 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.730293    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25073edb-2065-4a24-8ef1-3a6ed6c46b28-metrics-certs\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.753148 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.753082    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rfj\" (UniqueName: \"kubernetes.io/projected/25073edb-2065-4a24-8ef1-3a6ed6c46b28-kube-api-access-g8rfj\") pod \"maas-api-595cc648c-n96br\" (UID: \"25073edb-2065-4a24-8ef1-3a6ed6c46b28\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:27.809620 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:27.809593    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:28.111404 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:28.111378    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-n96br"]
Oct 29 23:47:28.113309 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:47:28.113283    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25073edb_2065_4a24_8ef1_3a6ed6c46b28.slice/crio-5282b4b15dc6e2ee97164db7ec33a2a9b61b0e78d98c9541c79a0f66abc9f30d WatchSource:0}: Error finding container 5282b4b15dc6e2ee97164db7ec33a2a9b61b0e78d98c9541c79a0f66abc9f30d: Status 404 returned error can't find the container with id 5282b4b15dc6e2ee97164db7ec33a2a9b61b0e78d98c9541c79a0f66abc9f30d
Oct 29 23:47:28.114549 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:28.114532    6203 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:47:28.921958 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:28.921914    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br" event={"ID":"25073edb-2065-4a24-8ef1-3a6ed6c46b28","Type":"ContainerStarted","Data":"5282b4b15dc6e2ee97164db7ec33a2a9b61b0e78d98c9541c79a0f66abc9f30d"}
Oct 29 23:47:32.947391 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:32.947355    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br" event={"ID":"25073edb-2065-4a24-8ef1-3a6ed6c46b28","Type":"ContainerStarted","Data":"4feefcdceb17b86baa330f9d85d47354df7611185933e30d5ee65752aa2be4a2"}
Oct 29 23:47:32.947911 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:32.947450    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:32.997959 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:32.997907    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br" podStartSLOduration=2.190675351 podStartE2EDuration="5.997873127s" totalImagesPullingTime="3.807197776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:47:32.996506555 +0000 UTC m=+2031.100766941" watchObservedRunningTime="2026-10-29 23:47:32.997873127 +0000 UTC m=+2031.102133553"
Oct 29 23:47:38.968235 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:38.968203    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-n96br"
Oct 29 23:47:39.923665 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:47:39.923628    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:47:47.743442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:47:47.743410    6203 scope.go:122] "RemoveContainer" containerID="bce47be3e69709af02dbb50c84bc1bcb7bff3b7d73894d3f8f6f1f3d9b6f0440"
Oct 29 23:48:09.924411 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:48:09.924374    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:48:17.975064 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:17.975026    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c8556598f-hglq5_7d144a1e-9b31-4af5-8b4f-47f1fab87d4e/authorino/0.log"
Oct 29 23:48:28.701128 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:28.701053    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-569f644b7c-nl8kw_42e769b0-8c1a-401c-8255-38bc5c4ba61d/manager/0.log"
Oct 29 23:48:30.079702 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:30.079670    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-46ppt_cb04d561-0325-4a58-bab5-4a255f081c2e/manager/0.log"
Oct 29 23:48:30.526631 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:30.526598    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57666cc899-7j9mq_36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c/copy-manifests/0.log"
Oct 29 23:48:30.570163 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:30.570130    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57666cc899-7j9mq_36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c/manager/0.log"
Oct 29 23:48:32.278422 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:32.278393    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-n96br_25073edb-2065-4a24-8ef1-3a6ed6c46b28/maas-api/0.log"
Oct 29 23:48:32.641973 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:32.641866    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-p9jwr_6a81917f-558d-4676-8767-67b502e6a4e2/revoke-keys/0.log"
Oct 29 23:48:32.837569 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:32.837534    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-hzdkl_210fbfef-0dbf-4d98-a293-7aac91094004/revoke-keys/0.log"
Oct 29 23:48:33.027796 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:33.027766    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-48a6af36-5pmsj_f3456cba-6951-4f2f-af14-b088ef79dd6c/revoke-keys/0.log"
Oct 29 23:48:33.397429 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:33.397347    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-7vzrp_04177281-b5d0-4300-b441-363f2e175ce6/revoke-keys/0.log"
Oct 29 23:48:33.574231 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:33.574206    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-8btxv_23798fc1-6abb-43b3-b20e-ef3e814ece08/revoke-keys/0.log"
Oct 29 23:48:33.766574 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:33.766544    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d74bef3b-k4h4v_a9458015-0a10-421f-bade-7fbb1c19893b/revoke-keys/0.log"
Oct 29 23:48:35.676463 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:35.676430    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c8556598f-hglq5_7d144a1e-9b31-4af5-8b4f-47f1fab87d4e/authorino/0.log"
Oct 29 23:48:36.949460 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:36.949418    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-mj82z_c162a52c-5579-4b27-84de-c7dfae18774b/manager/0.log"
Oct 29 23:48:38.152257 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:38.152226    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8578b44655-5b6lj_291aac22-e1e9-4172-85ae-865139758cc6/kube-auth-proxy/0.log"
Oct 29 23:48:38.540647 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:38.540615    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-7bzbg_90102371-81a6-4419-a23e-72f0ba01c595/istio-proxy/0.log"
Oct 29 23:48:39.923374 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:48:39.923333    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:48:40.142465 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:40.142431    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-rwn2k_9b2c909d-461e-4fff-9d3f-af4bc893adc0/istio-proxy/0.log"
Oct 29 23:48:40.345520 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:40.345442    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-vqnqf_bf9e8ad1-a722-4619-94b6-32d6e97002fc/istio-proxy/0.log"
Oct 29 23:48:42.084930 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:42.084891    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mbwdj_5c660259-6736-4175-a257-4e65889e0761/main/0.log"
Oct 29 23:48:42.661968 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:42.661936    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6_415d9077-00ae-4e22-8eed-8e1e0f94e1b1/main/0.log"
Oct 29 23:48:42.850290 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:42.850259    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2_8f451549-e817-46f9-bf9c-50f78d4c9091/main/0.log"
Oct 29 23:48:43.050088 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:43.050046    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk_1e8aa778-5206-47f8-8379-03976219cdcc/main/0.log"
Oct 29 23:48:43.968642 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:43.968610    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c8556598f-hglq5_7d144a1e-9b31-4af5-8b4f-47f1fab87d4e/authorino/0.log"
Oct 29 23:48:47.770713 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:47.770677    6203 scope.go:122] "RemoveContainer" containerID="be80b127ec230c20ba3a360eb25005dfe9c039ddace9f785e29418d29cfc4902"
Oct 29 23:48:47.779533 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:47.779507    6203 scope.go:122] "RemoveContainer" containerID="b0470cdd87a27fb606ee243b09f8d679cd9c9074d34eecbedb8bad093bf21205"
Oct 29 23:48:53.342756 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:53.342725    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-569f644b7c-nl8kw_42e769b0-8c1a-401c-8255-38bc5c4ba61d/manager/0.log"
Oct 29 23:48:54.662595 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:54.662563    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-46ppt_cb04d561-0325-4a58-bab5-4a255f081c2e/manager/0.log"
Oct 29 23:48:55.123936 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:55.123905    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57666cc899-7j9mq_36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c/copy-manifests/0.log"
Oct 29 23:48:55.167546 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:55.167518    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57666cc899-7j9mq_36ad64d8-da8d-40e7-b3fc-a0d28f2b8e2c/manager/0.log"
Oct 29 23:48:56.572641 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:56.572605    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-n96br_25073edb-2065-4a24-8ef1-3a6ed6c46b28/maas-api/0.log"
Oct 29 23:48:56.928851 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:56.928817    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-p9jwr_6a81917f-558d-4676-8767-67b502e6a4e2/revoke-keys/0.log"
Oct 29 23:48:57.110308 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:57.110280    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-hzdkl_210fbfef-0dbf-4d98-a293-7aac91094004/revoke-keys/0.log"
Oct 29 23:48:57.283993 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:57.283910    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-48a6af36-5pmsj_f3456cba-6951-4f2f-af14-b088ef79dd6c/revoke-keys/0.log"
Oct 29 23:48:57.660589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:57.660494    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-7vzrp_04177281-b5d0-4300-b441-363f2e175ce6/revoke-keys/0.log"
Oct 29 23:48:57.842734 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:57.842708    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-8btxv_23798fc1-6abb-43b3-b20e-ef3e814ece08/revoke-keys/0.log"
Oct 29 23:48:58.026565 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:58.026536    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d74bef3b-k4h4v_a9458015-0a10-421f-bade-7fbb1c19893b/revoke-keys/0.log"
Oct 29 23:48:59.589616 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:48:59.589585    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c8556598f-hglq5_7d144a1e-9b31-4af5-8b4f-47f1fab87d4e/authorino/0.log"
Oct 29 23:49:00.828105 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:00.828068    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-mj82z_c162a52c-5579-4b27-84de-c7dfae18774b/manager/0.log"
Oct 29 23:49:01.808777 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:01.808744    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8578b44655-5b6lj_291aac22-e1e9-4172-85ae-865139758cc6/kube-auth-proxy/0.log"
Oct 29 23:49:02.182247 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:02.182218    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-7bzbg_90102371-81a6-4419-a23e-72f0ba01c595/istio-proxy/0.log"
Oct 29 23:49:03.693060 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:03.693033    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-rwn2k_9b2c909d-461e-4fff-9d3f-af4bc893adc0/istio-proxy/0.log"
Oct 29 23:49:03.898812 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:03.898766    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-vqnqf_bf9e8ad1-a722-4619-94b6-32d6e97002fc/istio-proxy/0.log"
Oct 29 23:49:05.211684 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:05.211651    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mbwdj_5c660259-6736-4175-a257-4e65889e0761/main/0.log"
Oct 29 23:49:05.752625 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:05.752589    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b825sq6_415d9077-00ae-4e22-8eed-8e1e0f94e1b1/main/0.log"
Oct 29 23:49:05.942835 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:05.942802    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-r6vp2_8f451549-e817-46f9-bf9c-50f78d4c9091/main/0.log"
Oct 29 23:49:06.129022 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:06.128922    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-gwksk_1e8aa778-5206-47f8-8379-03976219cdcc/main/0.log"
Oct 29 23:49:08.028208 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:08.028174    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"]
Oct 29 23:49:08.042525 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:08.042487    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8btxv"]
Oct 29 23:49:09.923666 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:49:09.923625    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:49:10.016732 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:10.016699    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23798fc1-6abb-43b3-b20e-ef3e814ece08" path="/var/lib/kubelet/pods/23798fc1-6abb-43b3-b20e-ef3e814ece08/volumes"
Oct 29 23:49:24.031589 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:24.031548    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"]
Oct 29 23:49:24.046021 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:24.045993    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7vzrp"]
Oct 29 23:49:26.017800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:49:26.017760    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04177281-b5d0-4300-b441-363f2e175ce6" path="/var/lib/kubelet/pods/04177281-b5d0-4300-b441-363f2e175ce6/volumes"
Oct 29 23:49:39.923916 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:49:39.923850    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:50:09.924219 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:50:09.924130    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:50:17.029178 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:17.029142    6203 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"]
Oct 29 23:50:17.043442 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:17.043411    6203 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a6af36-5pmsj"]
Oct 29 23:50:18.017247 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:18.017213    6203 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3456cba-6951-4f2f-af14-b088ef79dd6c" path="/var/lib/kubelet/pods/f3456cba-6951-4f2f-af14-b088ef79dd6c/volumes"
Oct 29 23:50:31.094155 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:31.094122    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mkj2j_d9192aec-64e7-4d0f-8066-e236dc75b87d/global-pull-secret-syncer/0.log"
Oct 29 23:50:31.987514 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:31.987484    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-44xj8_faf7f273-a82a-40a5-811c-efb7281d1db6/konnectivity-agent/0.log"
Oct 29 23:50:32.546147 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:32.546116    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-224.ec2.internal_88f04efcc82b4a40de30b52c3590d78c/haproxy/0.log"
Oct 29 23:50:38.718732 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:38.718686    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c8556598f-hglq5_7d144a1e-9b31-4af5-8b4f-47f1fab87d4e/authorino/0.log"
Oct 29 23:50:39.550193 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:39.550140    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-mj82z_c162a52c-5579-4b27-84de-c7dfae18774b/manager/0.log"
Oct 29 23:50:39.922805 ip-10-0-139-224 kubenswrapper[6203]: E1029 23:50:39.922767    6203 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gkt7z" containerName="extractor"
Oct 29 23:50:40.755531 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.755493    6203 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"]
Oct 29 23:50:40.759393 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.759368    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.782737 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.782711    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"]
Oct 29 23:50:40.793255 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.793230    6203 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w8hzv\"/\"default-dockercfg-l6v7j\""
Oct 29 23:50:40.870972 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.870934    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w8hzv\"/\"openshift-service-ca.crt\""
Oct 29 23:50:40.871183 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.871013    6203 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w8hzv\"/\"kube-root-ca.crt\""
Oct 29 23:50:40.882079 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.882052    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-lib-modules\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.882211 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.882177    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-sys\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.882273 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.882205    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k9g7\" (UniqueName: \"kubernetes.io/projected/32c2c701-c2bb-4113-bcff-16bf69f0ec51-kube-api-access-7k9g7\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.882362 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.882288    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-proc\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.882362 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.882333    6203 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-podres\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983448 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983410    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-podres\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983455    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-lib-modules\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983513    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-sys\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983533    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k9g7\" (UniqueName: \"kubernetes.io/projected/32c2c701-c2bb-4113-bcff-16bf69f0ec51-kube-api-access-7k9g7\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983546    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-podres\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983562    6203 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-proc\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983639    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-lib-modules\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983677    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-proc\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:40.983909 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:40.983698    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32c2c701-c2bb-4113-bcff-16bf69f0ec51-sys\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:41.009461 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:41.009403    6203 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k9g7\" (UniqueName: \"kubernetes.io/projected/32c2c701-c2bb-4113-bcff-16bf69f0ec51-kube-api-access-7k9g7\") pod \"perf-node-gather-daemonset-vg2pw\" (UID: \"32c2c701-c2bb-4113-bcff-16bf69f0ec51\") " pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:41.071506 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:41.071472    6203 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:41.369180 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:41.369153    6203 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"]
Oct 29 23:50:41.370720 ip-10-0-139-224 kubenswrapper[6203]: W1029 23:50:41.370693    6203 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32c2c701_c2bb_4113_bcff_16bf69f0ec51.slice/crio-dc49d985907c2bab8954eec08b198dd79ad3fa78d76adc344bcdb99647f120d2 WatchSource:0}: Error finding container dc49d985907c2bab8954eec08b198dd79ad3fa78d76adc344bcdb99647f120d2: Status 404 returned error can't find the container with id dc49d985907c2bab8954eec08b198dd79ad3fa78d76adc344bcdb99647f120d2
Oct 29 23:50:41.930089 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:41.930050    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw" event={"ID":"32c2c701-c2bb-4113-bcff-16bf69f0ec51","Type":"ContainerStarted","Data":"9c0f4bfb0a5ac9e37b8bfbb979703da87921aee449cd59ed941c7e70ed9d7867"}
Oct 29 23:50:41.930285 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:41.930108    6203 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw" event={"ID":"32c2c701-c2bb-4113-bcff-16bf69f0ec51","Type":"ContainerStarted","Data":"dc49d985907c2bab8954eec08b198dd79ad3fa78d76adc344bcdb99647f120d2"}
Oct 29 23:50:41.930285 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:41.930146    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:41.976763 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:41.976712    6203 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw" podStartSLOduration=1.976698003 podStartE2EDuration="1.976698003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:50:41.975914249 +0000 UTC m=+2220.080174635" watchObservedRunningTime="2026-10-29 23:50:41.976698003 +0000 UTC m=+2220.080958389"
Oct 29 23:50:42.043102 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:42.043071    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-s657m_677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/kube-state-metrics/0.log"
Oct 29 23:50:42.161279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:42.161243    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-s657m_677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/kube-rbac-proxy-main/0.log"
Oct 29 23:50:42.288789 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:42.288692    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-s657m_677ae4b3-e7e3-4b5d-9eaf-d35a0df3b82f/kube-rbac-proxy-self/0.log"
Oct 29 23:50:42.552501 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:42.552416    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-t2sfs_d735fb1d-a5b9-4d8d-af87-bdaded8f82e6/monitoring-plugin/0.log"
Oct 29 23:50:43.078417 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:43.078387    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f24wt_71eca11c-bf4c-473c-b64a-64041bbe808d/node-exporter/0.log"
Oct 29 23:50:43.203056 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:43.203023    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f24wt_71eca11c-bf4c-473c-b64a-64041bbe808d/kube-rbac-proxy/0.log"
Oct 29 23:50:43.329419 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:43.329340    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f24wt_71eca11c-bf4c-473c-b64a-64041bbe808d/init-textfile/0.log"
Oct 29 23:50:43.851856 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:43.851807    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7mmsg_bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/kube-rbac-proxy-main/0.log"
Oct 29 23:50:43.977870 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:43.977842    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7mmsg_bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/kube-rbac-proxy-self/0.log"
Oct 29 23:50:44.104732 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:44.104646    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7mmsg_bfc1c0f0-73f0-4745-8e51-1d6aa2d14c53/openshift-state-metrics/0.log"
Oct 29 23:50:44.241551 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:44.241514    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8f43368-e2d8-4fca-8f89-e3319058b079/prometheus/0.log"
Oct 29 23:50:44.416605 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:44.416532    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8f43368-e2d8-4fca-8f89-e3319058b079/config-reloader/0.log"
Oct 29 23:50:44.560131 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:44.560106    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8f43368-e2d8-4fca-8f89-e3319058b079/thanos-sidecar/0.log"
Oct 29 23:50:44.691674 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:44.691645    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8f43368-e2d8-4fca-8f89-e3319058b079/kube-rbac-proxy-web/0.log"
Oct 29 23:50:44.823498 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:44.823466    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8f43368-e2d8-4fca-8f89-e3319058b079/kube-rbac-proxy/0.log"
Oct 29 23:50:44.949964 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:44.949861    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8f43368-e2d8-4fca-8f89-e3319058b079/kube-rbac-proxy-thanos/0.log"
Oct 29 23:50:45.077999 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:45.077967    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8f43368-e2d8-4fca-8f89-e3319058b079/init-config-reloader/0.log"
Oct 29 23:50:45.982636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:45.982606    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-866cccd46d-46pmg_7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/thanos-query/0.log"
Oct 29 23:50:46.108382 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:46.108353    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-866cccd46d-46pmg_7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/kube-rbac-proxy-web/0.log"
Oct 29 23:50:46.264279 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:46.264203    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-866cccd46d-46pmg_7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/kube-rbac-proxy/0.log"
Oct 29 23:50:46.391217 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:46.391189    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-866cccd46d-46pmg_7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/prom-label-proxy/0.log"
Oct 29 23:50:46.516253 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:46.516177    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-866cccd46d-46pmg_7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/kube-rbac-proxy-rules/0.log"
Oct 29 23:50:46.640952 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:46.640925    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-866cccd46d-46pmg_7e79e4e9-bd92-453f-bb0b-3c3dc213c6f3/kube-rbac-proxy-metrics/0.log"
Oct 29 23:50:47.811196 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:47.811163    6203 scope.go:122] "RemoveContainer" containerID="48b699b1167e6042dfd2e7571fcec8c9b07ab929b17e320fc71ed454f81437c5"
Oct 29 23:50:47.820090 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:47.820067    6203 scope.go:122] "RemoveContainer" containerID="75d6edff1d00f737fb1ccf6fd2a6b5f4a42a715c18d08c8dc846d9eee9fea200"
Oct 29 23:50:47.944800 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:47.944775    6203 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w8hzv/perf-node-gather-daemonset-vg2pw"
Oct 29 23:50:54.408636 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:54.408606    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kjc8k_1769c496-3b59-4aba-a674-13dae505ed09/dns/0.log"
Oct 29 23:50:54.533015 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:54.532984    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kjc8k_1769c496-3b59-4aba-a674-13dae505ed09/kube-rbac-proxy/0.log"
Oct 29 23:50:54.533846 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:54.533822    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8578b44655-5b6lj_291aac22-e1e9-4172-85ae-865139758cc6/kube-auth-proxy/0.log"
Oct 29 23:50:54.680112 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:54.680084    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-7bzbg_90102371-81a6-4419-a23e-72f0ba01c595/istio-proxy/0.log"
Oct 29 23:50:55.179976 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:55.179943    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zx5vd_89e02182-3b19-41c8-9040-f75a4254187b/dns-node-resolver/0.log"
Oct 29 23:50:55.206445 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:55.206415    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-rwn2k_9b2c909d-461e-4fff-9d3f-af4bc893adc0/istio-proxy/0.log"
Oct 29 23:50:55.277217 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:55.277184    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-746ff8f49f-vqnqf_bf9e8ad1-a722-4619-94b6-32d6e97002fc/istio-proxy/0.log"
Oct 29 23:50:56.628756 ip-10-0-139-224 kubenswrapper[6203]: I1029 23:50:56.628730    6203 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwc9k_96cf9d61-5fb4-45ed-b68b-868a6738f12c/node-ca/0.log"